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Abstract 

Attention  and  motivation  are  two  factors,  which 
are  important  for  motor  relearning  following 
stroke.  In  this  paper  we  introduce  and  present  the 
novel  concepts  for  the  use  of  haptics  technology  to 
deliver  therapy  to  patients  with  arm  impairments 
following  stroke.  The  results  of  the  ongoing  initial 
pilot  studies  have  shown  the  feasibility  of  the  system 
presented  here.  The  new  approach  will  potentially 
improve  the  patient’s  attention  and  motivation  and 
hence  enhance  therapy  effectiveness. 

Key  Words 

Motor  Relearning,  Robot  Mediated  Therapy, 
Virtual  Environments,  Perception  and  Haptic 
Feedback. 

1.  Introduction 

Stroke  is  the  most  common  cause  of 
acquired  physical  disability  with  estimates 
of  the  annual  incidence  of  stroke  ranging 
from  180  per  100,000  in  the  USA  to  200 
per  100,000  in  England  and  280  per 
100,000  in  Scotland  [1].  The  incidence 
rates  increase  with  age,  doubling  for  every 
10  years  after  the  age  of  45  [2]. 
Approximately  one-third  of  the  patients 
surviving  from  a  stroke  are  left  with  severe 
disabilities  [2].  Evidence  has  shown  that 
early  and  intensive  physiotherapy  improves 
patient  recovery  [3]  and  several  authors 


have  already  proposed  using  robots  to 
deliver  this  type  of  physiotherapy.  [4,5,6] 

In  the  GENTLE/S  project  (a  project 
under  the  quality  of  life  initiative  of 
framework  5  of  the  European  Commission 
to  evaluate  robot-mediated  therapy  in 
stroke  rehabilitation)  we  go  further  and 
propose  a  system  based  on  haptic 
technology.  Haptics  have  been  widely  used 
for  different  applications  ranging  from 
Surgery  Simulators  [7,8],  Virtual  Reality- 
Based  Telerehabilitation  systems  with 
force  feedback  [9]  to  3D  painting 
interfaces  [10]. 

In  the  context  of  haptics  and  force 
feedback  we  have  already  introduced  and 
implemented  a  novel  mathematical  model 
directed  to  deliver  a  system  capable  of 
correcting  movement  in  machine  assisted 
stroke  rehabilitation.  This  system  can  be 
used  for  errorless  learning  techniques  and 
intensive  rehabilitation  treatment  for 
patients  recovering  post-stroke.  It  allows 
force  feedback  to  be  delivered  to  the 
patient  arm  via  a  3DOF  haptic  device 
(Haptic  Master  from  Fokker  Control 
Systems)  strong  enough  to  correct  point  to 
point  movement  within  a  virtual/real  or 
augmented  task  oriented  therapy. 

Following  initial  input  from  members  of 
the  Young  Stroke  Association  at  Stoke-on- 
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Trent,  we  have  enhanced  movement 
realism  by  implementing  a  model  based  on 
the  dynamics  of  human  movement.  This 
model  allows  the  implementation  of 
realistic  force  feedback  in  curvilinear  3D 
movement  patterns  [11,  12]. 

We  are  investigating  new  ways  that  will 
potentially  improve  patient  attention  and 
motivation  and  therefore  enhance  therapy 
effectiveness.  The  following  sections  of 
this  paper  will  present  the  system 
developed,  the  implementation  of  novel 
theories  and  concepts  directed  to  the 
improvement  of  motivation  and  the  initial 
results  and  feedback  obtained  from  the 
initial  pilot  trials  with  hemiplegic  patients. 

2.  How  can  we  improve  patients’ 
motivation? 

Attention  and  motivation  are  two  factors, 
which  are  essential  for  brain  plasticity  [13]. 
Plasticity  is  the  ability  of  the  brain  to  re¬ 
organise  and  adapt  to  training  and 
experience.  This  ability  to  change  has 
therapeutic  benefit  as  the  brain  has  the 
potential  to  re-map  the  area  affected  by 
injury/stroke  in  response  to  training  of 
skilled  tasks  and  hence  allow  motor 
recovery.  Lack  of  motivation  has  been 
identified  as  a  major  cause  of  failure  to 
benefit  from  neurorehabilitation  [14].  It  is 
also  important  that  the  tasks  performed  by 
the  patient  are  meaningful  for  the  patient 
and  relate  to  their  perceived  problems  [15]. 

The  GENTLE/S  system  provides  the 
opportunity  for  repetitive  task  oriented 
movement,  a  form  of  therapeutic 
intervention  that  has  been  shown  to  be  of 
benefit  [16,17],  In  addition  to  this  it 
provides  the  following  feedback  to  the 
patient  to  enhance  attention  and  motivation 
and  hence  further  facilitate  motor  recovery 
through  brain  plasticity: 

•  Visual  Feedback  -  Realistic  and 
accurate  goal  oriented  3D 
environments.  This  can  be  anything 


from  a  virtual  room,  a  virtual  kitchen, 
museum,  to  an  interactive  game,  etc. 

•  Haptic  Feedback  -  Kinaesthetic 
feedback  helps  to  discriminate  physical 
properties  of  virtual  objects,  such  as 
geometry.  It  can  also  be  used  to  deliver 
physical  therapy  to  a  human  subject 
using  haptic  interfaces.  In  our  system  it 
is  used  to  guide  the  patients  arm  along 
a  pre-set  movement  pattern.  The 
amount  of  force  used  to  bring  the  arm 
back  to  the  set  path  can  be  varied  to 
suit  the  individuals  needs. 

•  Auditory  feedback  -  Encouraging 
words  and  sounds  are  played  when  the 
user  is  trying  to  perform  a  task, 
congratulatory  words  when  a  task  was 
achieved  with  success  and  comforting 
words  when  the  task  wasn’t  achieved. 

•  Performance  Feedback  -  Results  of 
previous  tasks  can  be  displayed  stating 
the  emors  committed  and  the  level  of 
help  obtained  to  complete  the  task. 

We  hope  to  be  able  to  foster  the  patients’ 
trust  by  developing  a  sense  of  friendship 
and  companionship  between  the  patient 
and  the  system.  A  personality  can  be  added 
to  the  system  by  having  a  character 
(wizard)  that  interacts  with  the  patient. 
Different  wizards  can  be  implemented  for 
different  personalities.  These  are  defined 
and  assigned  to  the  patient  once  the 
therapist  has  assessed  what  their  interests 
are.  We  hope  that  the  wizards  will  further 
enhance  motivation  and  attention  during 
this  therapy. 

3.  The  System 

The  current  prototype  system  (Fig.  1) 
consists  of  a  frame  with  two  chairs,  a 
shoulder  support  mechanism,  a  wrist 
connection  mechanism,  two  embedded 
computers,  a  large  computer  screen  with 
speakers,  an  exercise  table,  a  keypad  and  a 
3DOF  haptic  interface  arm  (Haptic 
Master).  The  patient  is  seated  on  the  chair 
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with  their  arm  positioned  in  an  elbow 
orthosis  suspended  from  the  overhead 
frame.  This  is  to  eliminate  the  effects  of 
gravity  and  address  the  problem  of 
shoulder  subluxation.  The  wrist  is  placed 
on  a  wrist-orthosis  connected  to  the  robot 
arm  using  a  quick  release  mechanism  (Fig. 
2). 


Figure  1  -  GENTLE/S  Prototype  1 


Initial  studies  with  a  group  of  stroke 
patients  at  the  Young  Stroke  Association  in 
Stoke-on-Trent,  UK,  have  encouraged 
further  development  and  enhancement  of 
Virtual  Environments  to  deliver  therapy 
[18].  Virtual  tasks  consist  of  a  goal 
oriented  computer  graphical  user  interface 
based  on  virtual  reality  visualisation 
techniques.  The  therapist,  using  the  task 
editor,  can  create  a  virtual  task.  The  system 
allows  the  therapist  to  create  new 
movement  patterns  in  virtual  3D 
environments  and  to  define  how  objects 
are  related  to  each  other  and  what  kind  of 
behaviour  they  represent  to  the  patient.  For 
instance,  one  simple  activity  involves  the 
patient  to  reach  for  a  can  of  Pepsi  and 
move  it  close  to  a  book  in  the  table. 


Figure  2  -  Robot  end-effector  and  arm 
orthosis 


4.  Initial  pilot  trials 

Following  ethical  and  safety  approval, 
the  initial  pilot  trials  are  ongoing.  We  have 
created  and  used  3  different  graphical 
environments  (Fig.  3)  for  the  initial  trials: 

1.  Empty  room  -  A  graphically  poor 
environment  consisting  of  white  walls, 
a  floor  and  ceiling,  representing  the  3D 
workspace  of  the  system. 

2.  Real  room  -  An  environment  that 
resembles  what  the  patient  sees  on  the 
table  in  the  real  world.  The  mat  with  4 
different  shapes  that  is  on  the  table  is 
represented  in  the  3D  graphical 
environment. 

3.  Joaquim’s  room  -  A  high  detail  3D 
environment  of  a  room  comprising  of 
a  table,  several  objects  (a  book,  can  of 
Pepsi),  portrait  of  a  baby,  window, 
curtains,  etc. 


Figure  3  -  (A)  Empty  room,  (B)  Real  room, 
(C)  -  Joaquim’s  room 

We  have  assessed  the  suitability  and 
usefulness  of  the  three  different  visual 
environments,  the  2  levels  of  force 
feedback  and  the  motivational  aspects  of 
the  system  using  a  questionnaire  survey  of 
patients  with  stroke  using  the  first 
prototype  in  the  pilot  trials. 

4.1  Perception 

Several  questions  were  asked  to  evaluate 
the  patients’  ability  to  identify  and 
correlate  his  movements  in  the  real  world 
to  what  is  displayed  on  the  screen.  This 
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information  will  help  us  to  design 
environments  that  are  easy  to  interact  with. 
Patients  completed  a  simple  4-point 
exercise  (3D  pyramid)  in  each  of  the  three 
environments  for  2  minutes.  Four 
questions  were  asked  to  evaluate  their 
opinion  on  the  environment’s  look, 
realism,  and  ease  of  use. 

4.2  Force  feedback 

The  Kinaesthetic  feedback  forces  were 
used  on  2  different  settings:  strong  (i.e. 
highly  stiff;  exerted  a  strong  pull  back  to 
the  desired  path)  and  weak  (i.e.  low 
stiffness;  allowed  the  patient  to  deviate 
more  from  the  desired  path  and  exerted  a 
lesser  force  to  bring  the  limb  back  to  the 
path). 

The  patients  exercised  for  three  minutes 
on  each  setting  in  environment  A  (fig.  2 
(A)).  To  evaluate  the  effectiveness  of  Force 
feedback,  questions  were  asked  to  collect 
patient’s  opinion  on  whether  the  force 
feedback  helped  them  to  perform  the 
movement,  which  setting  made  it  easiest  to 
complete  the  movement  and  which  setting 
they  preferred. 

4.3  Motivation 

These  tests  were  performed  to  collect 
patient’s  opinion  on  motivational  aspects 
of  the  interaction  between  people  with 
stroke  and  the  haptic  interface.  We 
collected  information  on  the  total  time  that 
they  exercised  for,  whether  exercising  in 
the  system  was  more  interesting  than  their 
normal  therapy  and  how  long  they  thought 
they  could  exercise  for  before  it  would 
become  boring. 

After  10-minute  intervals,  patients  were 
asked  whether  they  wished  to  continue 
with  the  same  exercise,  change  to  another 
exercise  or  stop. 

5.  Results  and  discussion 

The  tests  described  in  the  previous 
section  were  performed  by  8  respondents, 


of  which  1  had  a  Cardio  Vascular  Accident 
(CVA)  more  than  one  year  ago,  3  had  a 
CVA  between  3  months  and  1  year  and  the 
remaining  3  had  a  CVA  less  than  three 
months  prior  to  testing.  Of  these,  6  had  left 
hemiplegia  and  2  right  hemiplegia,  6  were 
female  and  2  were  male.  Their  ages  ranged 
from  50  to  81  years. 

5.1  Perception 

Table  1  shows  that  although  most 
patients  found  environment  C  the  nicest  to 
look  at,  6  of  the  8  patients  actually  enjoyed 
practising  in  the  mixed  environment  (B). 
This  suggests  that  by  providing  a  degree  of 
realism  we  are  correlating  what  the  eye 
actually  sees  on  the  screen  with  what  exist 
in  the  real  world,  which  in  turn  makes  the 
experience  more  enjoyable  for  the  patient. 

On  the  other  hand,  the  relatively 
complicated  graphical  environments  were 
not  felt  by  the  patients  to  be  the  easiest  in 
which  to  interpret  the  required  direction  of 
movement  from  the  3D  image. 


Ouestion 

Environments 

A 

B 

c 

Undecided 

1.  Which  room  look  the 
nicest? 

1 

3 

4 

2.  Which  room  felt  more 
realistic? 

0 

5 

3 

3.  In  Which  room  was 
easiest  to  know  where 
you  had  to  move  next? 

3 

2 

2 

1 

4.  Which  room  did  you 
enjoy  exercising  the 
most? 

2 

6 

0 

Table  1  -  Patient  answers  on  perception. 


5.2  Force  Feedback 

For  this  test  it  was  required  that  the 
patient  had  some  level  of  activity  and  were 
exercising  in  active  mode  (i.e.  the  robot  did 
not  provide  any  assistance  in  the 
completion  of  the  required  movement  and 
provided  a  velocity  dependant  resistance  to 
the  patients  effort). 

Four  patients  completed  this  section  all  of 
whom  responded  that  the  force  feedback 
was  useful  in  completing  the  required 
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movement  pattern.  Three  of  the  4  patients 
preferred  the  2nd  setting  (weak  force).  They 
commented  that  this  was  because  on  the 
strong  setting  they  had  to  work  harder. 
Although  this  may  be  less  enjoyable  it  may 
actually  be  of  greater  therapeutic  benefit 
and  it  is  worth  noting  that  2  of  the  4 
patients  had  a  better  quality  of  movement 
on  the  strong  setting.  We  therefore 
conclude  that  it  is  important  to  set  the  force 
to  accommodate  the  patients  needs. 

5.3  Motivation 

All  of  the  six  patients  that  completed  this 
section  felt  that  the  system  helped  them  to 
exercise  for  longer  than  they  normally 
would  in  their  traditional  therapy  session. 
Five  of  the  six  felt  that  therapy  on  the 
GENTLE/S  system  was  more  interesting 
than  their  traditional  therapy  session. 


Figure  4  -  Exercise  time  results. 


When  they  were  asked,  how  long  they 
could  exercise  for  before  it  would  become 
boring;  5  of  the  6  patients  responded  with 
times  that  were  greater  than  the  time  that 
they  had  spent  exercising  (Fig.  4).  It  is 
worth  noting  that  7  of  the  8  patients  in  the 
pilot  trial  stopped  exercising  because  of 
arm  fatigue  rather  than  boredom. 

6.  Future  Work 

The  initial  clinical  trial,  which  is 
currently  taking  place,  involves  evaluation 
of  the  prototype  in  terms  of  design,  patient 
comfort,  ergonomics  and  ease  of  use  by 
both  patient  and  therapist.  Later  in  the  year 
a  full  clinical  trial  will  commence,  which 


will  evaluate  the  GENTLE/S  system  in 
terms  of  therapeutic  benefit  at  the  levels  of 
impairment,  disability  and  handicap. 

Also  we  are  going  to  look  at  virtual  tasks 
allowing  manipulation  of  objects  and  the 
haptics  of  manipulation  of  virtual  and  real 
objects  in  real  tasks.  At  that  point  we  will 
investigate  and  analyse  the  biomechanical 
data  to  determine  if  haptic  technology  and 
the  development  of  wizard  intelligence  will 
positively  contribute  towards  the 
advancement  of  assistive  technologies. 

7.  Conclusion 

This  paper  has  presented  the  initial  ideas 
and  concepts  for  adapting  and  using 
dedicated  haptic  interfaces  as  part  of  the 
rehabilitation  process  of  stroke  patients. 
Our  approach  is  based  on  the  fact  that 
attention  and  motivation  are  the  keys  for 
recovery. 

The  initial  pilot  studies  have  shown  that 
the  majority  of  the  patients  were  positive 
towards  the  use  of  visual  and  haptic  cues. 
Some  of  the  results,  with  emphasis  on 
visual  perception,  suggested  that  we  would 
benefit  by  using  augmented  (mix) 
environments  and  also  we  would  need  to 
manipulate  objects  on  both  worlds. 

Our  force  feedback  test  suggests  that  a 
patient  might  feel  less  force  is  more  useful, 
although  this  does  not  necessarily  means 
that  it  is  therapeutic.  Further  investigation 
is  necessary  to  prove  the  usefulness  of 
haptic  interfaces,  especially  force  feedback. 

Also  this  result  allowed  us  to  conclude 
that  the  amount  of  assistive  force  should  be 
tuned  for  each  individual,  otherwise  it 
might  create  extra  impedance. 

Finally,  as  a  whole  the  system  has  proven 
to  motivate  hemiplegic  patients  to  exercise 
for  longer  than  they  normally  would  in 
their  traditional  therapy  session. 


5 


Eurohaptics  2001  Birmingham  July  2001 


Acknowledgements 

The  authors  are  pleased  to  acknowledge 
funding  from  the  European  commission 
quality  of  life  programme  framework  5. 

We  arc  grateful  to  all  our  colleagues  in 
the  GENTLE/S  consortium  for  their 
ongoing  commitment  to  this  work. 

References 

[1]  Isard,  P.  A.,  and  Forbes,  J.  F.  (1992).  The  cost  of 
stroke  to  the  National-Health-Service  in  Scotland. 
Cerebrovascular  Diseases.  2:  47-50. 

[2]  Abrams,  W.  B.,  and  Berkow,  R.  (1996  -  1997).  Merk 
Manual  of  Geriatrics.  NJ,  Merk  Research 
Laboratories. 

[3]  Carr,  J.  H.,  and  Shepherd,  R.  B.  (1987).  A  motor 
relearning  programme  for  stroke,  Oxford: 
Butlerworth  Heinemann. 

[4]  Krebs,  H.  I„  Hogan,  N„  Volpe,  B.  T.,  Aisen,  M.  L„ 
Edelstein,  L.,  and  Diels,  C.  (1999).  Robot-Aided 
neuro-Rehabilitation  in  Stroke:  Three-Year  Follow¬ 
up.  ICORR  -  Sixth  International  Conference  on 
Rehabilitation  Robotics,  Stanford,  CA. 

[5]  Reinkensmeyer,  D.  J.,  Schmit,  B.  D.,  and  Rymer,  W. 
Z.  (1999).  Assessment  of  active  and  passive  restraint 
during  guided  reaching  after  chronic  brain  injury. 
Annals  of  Biomedical  Engineering  27(6):  805-814. 

[6]  Lum,  P.  S.,  Burgar,  C.  G.,  Kenney,  D.  E.,  and  Van 
der  Loos,  H.  F.  M.  (1999).  Quantification  of  force 
abnormalities  during  passive,  and  active-assisted 
upper-limb  reaching  movements  in  post-stroke 
hemiparesis.  IEEE  Transactions  on  Biomedical 
Engineering  46(6):  652-662. 

[7]  Loureiro  R.C.V.  (1999).  MSc  Dissertation  in  A 
Laparoscopic  Virtual  Surgery  Simulator. 
Department  of  Electronic  and  Electrical 
Engineering,  University  of  Salford. 

[8]  Salleh,  R.,  Loureiro,  R.,  and  Caldwell,  D.  (2000). 
Haptic  feedback  for  VR  based  minimally  invasive 
surgical  (MIS)  training.  EUREL  -  European 
Advanced  Robotics  Systems  Master  Class  and 
Conference,  Salford,  Manchester,  UK. 

[9]  Popescu,  V.G.,  Burden,  G.C.,  Bouzit,  M„  and  Hentz, 
V.R.  PC-based  Telerehabilitation  System  with  force 
feedback.  Proceedings  of  MMR  7,  IOS  Press, 
Amsterdam,  Vol.  62,  pp.  261-267. 

[10]  Gregory,  A.,  Ehamann,  S.,  and  Lin,  M.C.  (1999). 
inTouch:  Interactive  multiresolution  modelling  and 
3D  painting  with  a  haptic  interface.  Technical 
report,  Department  of  Computer  Science,  University 
of  North  Carolina. 


[11]  Amirabdollahian,  F.,  Loureiro,  R.,  Driessen,  B.,  and 
Harwin,  W.  (2001).  Error  Correction  Movement  for 
Machine  Assisted  Stroke  Rehabilitation,  Integration 
of  Assistive  Technology  in  the  Information  Age, 
Mounir  Mokhtari  (Ed.),  IOS  Press,  Vol.  9,  pp.  60- 
65. 

[12]  Loureiro,  R.,  Amirabdollahian,  F.,  Driessen,  B.,  and 
Harwin,  W.  (2001).  A  novel  method  for  computing 
natural  path  for  robot  assisted  movements  in 
synthetic  worlds.  AAATE  -  6th  European 
conference  for  the  advancement  of  assistive 
technology,  Ljubljana,  Slovenia.  (Accepted) 

[13]  Robertson,  I.H.,  and  Murre,  J.M.J.  (1999). 
Rehabilitation  of  Brain  Damage:  Brain  Plasticity 
and  Principles  of  Guided  Recovery,  Psychological 
Bulletin  125(5):  544-7. 

[14]  Macciochi,  S.N,  Eaton,  B.  (1995).  Decision  and 
attribution  bias  in  neurorehabilitation.  Arch  Phys 
MEd  Rehabil  76,  521-4. 

[15]  Prigitano,  G.P.  (1999).  In  Cognitive 
NeuroRehabilitation,  Ed  Stuss  DT,  Winocur  G, 
Robertson  IH,  Chapter  15,  Cambridge  University 
Press,  Cambridge. 

[16]  Hesse,  S.,  Bertelt,  C.,  Jahnke,  M.T.,  Schaffrin,  A., 
Baake,  P.,  Malezic,  M.,  Mauritz,  K.H.  (1995). 
Treadmill  training  with  partial  body  weight  support 
compared  with  physiotherapy  in  non-ambulatory 
hemiparetic  patients.  Arch  Phys  Med  Rehabil  75(10) 
1087-93. 

[17]  Butefisch,  C.,  Hummelsheim,  H.,  Denzler,  P., 
Mauritz,  K.H.  (1995).  Repetitive  training  of  isolated 
movements  improves  the  outcome  of  motor 
rehabilitation  in  the  centrally  paretic  hand.  Journal 
Neurological  Sciences,  13,  p59-68. 

[18]  Graves,  M.  (2001),  Changing  lives  with  the  gentle 
touch  -  Robot  technology  could  mean  a 
breakthrough  for  people  disabled  by  strokes. 
Disability  Times  -  Technology,  March  2001. 


6 


Eurohaptics  2001  Birmingham  July  2001 


A  Fast  Finite  Element  Modelling  Tool  for  Surgical  Simulation 


Naoufel  AZOUZ 

Centre  d’Etudes  de  Mecanique  d’l-D-F 
Laboratoire  des  systemes  complexes 
Universite  d’Evry  Val  d’Essonne 
40  rue  du  Pelvoux,  91020  Evry,  FRANCE. 
E-mail :  azou7@cemif.univ-evry.fr 


ABSTRACT 

This  paper  present  an  algorithm  based  on 
finite  element  modelling  of  3D  solid  volumetric, 
suitable  for  real-time  animation  and  for  haptic 
interaction,  particularly  for  surgical  simulator.  The 
techniques  used  now  by  most  surgical  simulators 
are  based  on  approximate  methods  that  do  not 
represent  an  accurate  behaviour  of  the  deformable 
bodies.  The  finite  element  method  (FEM)  is  an 
alternative  because  of  its  accuracy,  but  it  is 
computationally  expensive.  In  this  paper  we  address 
this  problem  by  proposing  the  notion  of  e-mesh 
radius  in  analogy  with  the  similar  notion  used  in  the 
theory  of  adaptive  meshing. 

Keywords  :  Modelling,  Finite-Element,  Surgical 
Simulation,  e-mesh  radius. 

1.  INTRODUCTION 

The  surgical  simulation  becomes  possible, 
through  the  improvement  of  the  computers,  and 
some  special  purpose  mechanisms,  namely  haptic 
interface.  Various  graphical  based  environment  are 
being  identified  where  the  user  is  able  to  interact 
with  such  virtual  environment.  In  these 
environments,  the  user  manipulates  graphical 
objects  by  using  the  hand  interface  mechanism.  If 
the  interference  of  the  graphical  object  is  detected, 
desired  reaction  forces  are  generated  at  the  hand^ 
input  device  which  gives  the  user  a  sense  of 
physical  interaction  with  the  virtual  objects. 

Current  literature  in  the  area  of  haptic 
rendering  focuses  on  the  development  of  fast 
collision  detection  algorithm  and  some  basic 
contact  profiling  [1].  For  modelling  the  interaction 
with  the  soft  tissues  different  approaches  have  been 
investigated  [2], 
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Some  authors  proposed  modelling  approaches 
based  on  the  spring-mass  method  because  of  their 
simplicity  and  their  low  cost  computational  [3,4], 
However,  such  current  approaches  remain  to  be 
imprecise.  On  the  other  hand,  The  finite  element 
method  (FEM)  offers  an  alternative.  The  main 
problem  in  this  method  is  the  high  computational 
time. 

Cotin  &  al.  [5]  tried  to  overcome  this  problem 
by  using  a  pre-processing  technique  and  the 
condensation  method.  Berkley  and  Weghrost  [6] 
proposed  to  use  the  technique  of  the  banded 
matrix.  They  also  proposed  to  incorporate  the 
whole  volume  of  the  tissue  and  organs  for 
generating  the  finite  element  mesh. 

This  paper  is  concerned  with  determining  a 
suitable  method  for  defining  the  contact  profile  for 
constrained  objects.  The  objective  of  the  paper  is  to 
define  suitable  size  and  number  of  local  grid-mesh 
which  can  represent  a  suitable  physical  recreation 
model  for  the  contact  profile  with  realistic 
deformation.  For  this  purpose  a  number  of 
comparisons  between  the  traditional  . generation  of 
the  finite-element  mesh  and  a  new  notion  of  e- 
mesh  radius  is  made  especially  in  the  context  of 
piercing  and  cutting  human  tissue.  As  a  result  this 
study  offers  an  approach  to  reduce  the  computation 
time  for  determining  the  contact  profile  using  the 
notion  of  finite-element.  In  addition,  such  new 
approach  can  be  made  adaptive  where,  as  a 
function  of  interference  parameters  and  location  of 
the  contact  between  bodies,  local  mesh  can  be 
generated.  This  mesh  profile  can  be  defined 
optimal  for  computational  purposes  in  haptic 
rendering.  When  contact  is  broken  with  the  virtual 
object,  the  object  representation  will  have  its 
original  coarse  representation  of  mesh  defined 
through  CAD  environment  (or  generated  through 
MRI  data-sets).  The  prototypes  of  the  analysis  of 
this  study  are  developed  using  general  purpose 
finite  element  environment  (CASTEM  2000). 
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2.  Overview 

2.1  Mathematical  formulation 

In  this  section  We  briefly  introduce  the  theory  of 
elasticity  and  the  finite  element  method. 

Let  us  consider  a  body  (Q)  having  a  volume  V  and 
a  boundary  (dQ) .  At  first  approach,  we  consider 

that  the  object  has  a  linear  elastic  behaviour.  It  is 
subject  to  volumic  and  boundary  forces. 


From  the  linear  elasticity,  the  3x3  strain  tensor  V, 
of  the  body  subject  to  a  given  force  could  be  written 
as  a  function  of  the  displacement  U  as  : 

£  =  I(vu+vtu)  (l) 

V  is  the  gradient  operator. 

Let  us  note  a  the  3x3  symmetric  stress  tensor 
and  fv  the  volumic  forces  applied  on  the  body  .  In 
the  case  of  elastic  behaviour,  the  relation  between 
stress  tensor  and  strain  one  is  : 

a  =  E.e  (2) 

E  is  the  four  order  tensor  of  the  elasticity 
coefficients,  it  depends  on  the  mechanical 
characteristics  of  the  body  (Q). 

When  using  the  principle  of  virtual  works,  one  can 
obtain,  for  a  virtual  increment  of  displacement  8U  : 

-£a.5sdV  +  JQtn.5UdS  +  J,fySUdv  =  0  (3) 

t„  is  the  stress  vector  applied  on  the  boundary 

(3Q). 

To  overcome  the  complexity  of  modelling  objects 
based  on  continuum  mechanics,  i.e.  solving 
equation  (3),  the  body  (Q)  is  now  divided  into 
different  elementary  domains  limited  by  nodes. 
The  displacement  of  a  given  point  of  the  body 


could  be  approximate  as  a  function  of  the  nodal 
displacements  as  [7]  : 

U  =  [N]r.U  (4) 

[N]  is  the  matrix  of  interpolation  functions  and 
U  is  the  displacement  column  matrix  of  the  nodes. 
Noting  the  global  stiffness  matrix  of  the  form  : 

[K]=  J(D[N])t.E.D[N]  dV  (5) 

V 

D  is  a  differential  operator.  The  static  equation  (3) 
becomes  for  the  discretized  body  : 

[K].U  =  F  (6> 

F  here  represent  the  column  matrix  of  the  whole 

volumetric  and  boundary  forces  applied  on  the 
body. 

2.2  Multigrid 

In  the  minimally  invasive  surgery,  that  we  propose 
to  simulate,  the  region  influenced  by  the  action  of 
the  surgeon  tool  is  localised  in  the  vicinity  of  the 
contacting  point.  The  reaction  force  induced  by 
this  movement  is  mainly  depending  on  this  limited 
zone.  To  avoid  solving  equation  (6)  with  a  great 
number  of  degrees  of  freedom,  we  divide  the  object 
into  a  number  of  volumes.  In  the  neighbourhood 
of  the  contacting  point,  we  generate  fine  mesh.  We 
define  the  size  of  this  local  mesh  by  the  e-mesh 
radius  which  represent  a  characteristic  length  in 
this  volume.  For  the  remaining  of  the  volume,  we 
use  coarse  mesh.  In  classical  FEM,  this  procedure 
should  be  accompanied  by  the  minimisation  of  the 
errors  [7]  in  the  energy  norm  in  relation  with  the 
validity  of  the  FEM  approximation. 


Figure  2.  Modelling  of  the  cutting  problem 

In  haptic  rendering,  we  have  fewer  constraints 
regarding  the  accuracy  of  the  computation  of  the 
reaction  force.  In  order  to  study  this  hypothesis,  we 
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will  study  through  simulation  the  ratio  between  the 
reaction  force  given  by  our  modelling  method 
using  e-mesh  and  the  one  obtained  through  a  full 
order  3D  mesh  representing  the  “exact”  solution. 
The  example  selected  correspond  to  the  simulation 
of  the  cutting  problem  (figure  2). 

3.  Simulation  results 

In  order  to  determine  the  relationship  between  the 
full  order  3D  mesh  and  the  s-mesh  radius 
technique,  we  impose  a  given  displacement  (of 
piercing  or  cutting)  to  the  body  and  determine  the 
force  needed  for  such  task.  This  in  general  is  called 


different  parameters  of  the  study.  Other 
simulations  show  that  the  ratio  F/FnKX  could  be 
considered  as  constant  for  different  geometric  and 
mechanical  characteristics  of  an  elastic  body. 
Consequently  we  can  develop  a  relationship 
between  the  results  given  by  a  full  order  3D  mesh 
issued  from  an  off-line  simulation,  and  those  given 
by  s-mesh  simulation.  On  the  other  hand,  the 
computing  time  decreases  significantly  (about  50 
times  lesser)  from  the  computation  with  a  full 
order  3D  mesh  to  the  computation  with  the  “s- 
mesh  radius”  technique. 

4.  Conclusion 


inverse  problem . 

We  purpose  an  algorithm  where  we  can  change 
different  parameters  in  the  simulations  and 
compute  the  ratio  between  the  “real”  reaction  force 
computed  using  the  full-order  finite-element  model 
and  the  proposed  e-mesh  procedure. 

1 .  Input  (geometrical  shape,  mechanical 
characteristics,  types  of  mesh) ; 

2.  Input  (locations  of  concentration, 

geometry  of  the  probe); 

3.  do  ^-(calculate  the  real  reaction  force  to  an 
imposed  displacement  of  piercing  or  cutting); 

4.  do  <r  (calculate  the  reaction  force  with  a 
multigrid  referenced  by  an  e-mesh  radius); 

5.  do  (recalculate  the  reaction  force  with  a 
new  e-mesh  radius  till  the  global  volume  of 
the  body); 

6.  back  to  1  (with  new  parameters); 

end; 

Figure  3  shows  an  example  of  our  study.  The  plots 
represent  the  ratio  between  the  reaction  force  given 
by  the  e-mesh  procedure  and  the  one  obtained  by 
a  full  order  3D  mesh  (F/Fmm)  for  different  values  of 
the  poisson  ratio. 


The  figure  shows  that  for  a  relatively  small  E-mesh 
radius  (10mm) ,  we  obtain  more  than  80%  of  Fm^. 
This  ratio  remain  quasi-constant  when  we  change 


We  present  a  method  for  the  simulation  of  linear 
elastic  deformable  bodies  in  surgical  applications. 
We  proposed  and  are  developing  the  new  notion  of 
s-mesh  radius,  and  have  shown  that  it  is  a  useful 
tool  due  its  local  interaction  model,  accuracy,  and 
simplicity  of  implementation.  Preliminary  study 
shows  that  the  results  are  close  to  those  of  a  full 
order  3D  mesh  within  a  reasonable  radius  from  the 
contact  area. 

The  main  contribution  of  the  paper  is  to  determine 
a  guideline  for  developing  physically  based  models 
of  the  object  based  on  finite  element  method.  The 
guideline  suggest  an  approach  for  determining  the 
size  and  number  of  elements  involved  for  creating 
a  realistic  reaction  force  calculations  which  can  be 
used  in  haptic  rendering.  The  trade-offs  are 
between  the  realistic  sense  of  deformation  and  the 
computational  costs. 
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Abstract 

The  efficacy  of  virtual  reality-based  rehabilitation  for 
orthopedic  patients  needs  to  be  determined.  This  paper- 
presents  three  case  studies  in  which  the  use  of  the  Rutgers 
Ankle  haptic  interface  for  ankle  rehabilitation  is 
evaluated.  The  Rutgers  Ankle  is  a  computer-controlled 
compact  robotic  platform  that  measures  foot  position  and 
orientation  and  provides  resistive  forces  and  torques,  It 
has  been  integrated  with  a  PC  running  an  airplane 
piloting  VR  exercise  and  a  custom  electronic  control  box. 
The  system  underwent  pilot  clinical  trials  in  an  outpatient 
clinic  in  the  State  of  New  Jersey  (USA).  The  goals  of  the 
pilot  tests  were  to  determine  how  well  the  current  design 
of  the  interface  worked  in  a  clinical  setting  and  to  provide 
preliminary  evidence  of  efficacy  ( whether  patients 
benefited  from  incorporating  the  RA  into  their 
rehabilitation  program).  Results  on  three  patients 
participating  in  the  study  showed  improvements  in  range 
of  motion,  torque  generation  capacity  and  ankle 
mechanical  work  over  six  rehabilitation  sessions. 
Objective  measures  obtained  by  the  Rutgers  Ankle  show 
improvement  in  task  accuracy  to  1007c  for  Case  1,  a  five¬ 
fold  increase  in  ankle  power  output  for  Case  2,  and  a 
threefold  increase  for  Case  3.  Both  Case.  2  and  Case  3 
reached  1007c  task  accuracy  by  the  end  of  the  two-week 
rehabilitation  training.  Results  from  these  case  studies 
are  being  used  in  software  enhancements  and 
construction  of  a  dual-platform  system. 

1.  Introduction 

Physical  rehabilitation  may  benefit  from  the  recent 
advances  in  virtual  reality  (VR)-based  simulations.  VR 
exercises  can  be  used  to  motivate  patients  by  providing  an 


engaging,  multi-modal,  interactive  rehabilitation 
environment  [2,3].  In  order  to  allow  physical 
rehabilitation,  such  as  the  one  needed  by  orthopedic  and 
stroke  patients,  physical  interaction  with  rehabilitation 
devices  are  involved.  If  the  rehabilitation  is  done  in  virtual 
environments,  new  types  of  haptic  input/output  interfaces 
need  to  be  created.  One  example  is  the  Computer  Assisted 
Rehabilitation  Environment  (CAREN)  developed  in 
Holland  [1],  It  uses  a  computer-controlled  six-degree-of- 
freedom  large  platform  to  destabilize  the  patient  standing 
still  on  it.  Vision  tracking  is  used  to  measure  the  patient’s 
response  for  evaluation  purposes. 

Another  example  of  a  platform-based  computer 
interface  used  in  rehabilitation  is  the  Rutgers  Ankle 
(RA)[5],  The  RA  is  designed  for  patients  that  need  ankle 
and  knee  rehabilitation.  It  has  been  applied  in  the 
rehabilitation  of  patients  with  orthopedic  (ankle  sprain) 
[6]  and  stroke  diagnoses  [4]. 

This  paper  describes  first  the  RA  hardware  and 
software  system.  Subsequently  the  outcomes  of  its 
introduction  into  a  working  clinic  are  presented.  We 
describe  three  case  studies  of  patients  with 
musculoskeletal  ankle  injuries  of  different  etiologies. 
Interpretation  of  the  results  and  directions  for  future 
research  conclude  this  paper. 

2.  Virtual  Reality  system 

The  Rutgers  Ankle  system  consists  of  a  Stewart 
Platform-based  interface,  its  electro-pneumatic  control 
box,  a  PC  host  computer  and  a  mini-compressor,  as  shown 
in  Figure  1  [6]. 

The  platform  has  dual-acting  low-friction  piston 
actuators,  installed  in  parallel  with  linear  potentiometers 
used  as  position  sensors.  A  3-D  force  sensor  placed  under 
a  foot  attachment  measures  the  real-time  forces  and 
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torques  applied  by  the  mobile  platform.  The  work 
envelope  of  the  mobile  platform  accommodates  the 
normal  ankle  motion  for  a  sitting  patient.  Its  force  output 
depends  on  the  pressure  in  the  pistons,  up  to  a  maximum 
100  psi. 


Figure  1.  The  Rutgers  Ankle  System  [6],  ©  Rutgers 
University.  Reprinted  by  permission. 

The  control  interface  uses  peizo-electric  valves  that 
regulate  the  pressure  in  the  actuator  chambers  at  a 
bandwidth  of  500  Hz.  Low-level  servo  control  is 
performed  by  an  embedded  Pentium  board  (266  MHz), 
based  on  pressure  sensor  information  from  the  valve 
outputs.  The  servo  control  is  a  user-selectable  position  or 
force  control  loop,  which  runs  much  faster  than  would  be 
possible  on  the  PC  host  computer.  Communication  with 
the  PC  host  is  done  over  an  RS232  serial  line.  The  host 
PC  is  a  single-processor  Pentium  300  MHz  computer 
running  Windows  NT.  It  displays  the  VR  graphics,  user 
graphical  user  interface  (GUI),  and  stores  data  in  an 
Oracle  database 

3.  Exercise  software 

The  exercise  software  was  developed  using  the 
WorldToolKit  software  library  for  real-time  graphics  and 
the  Oracle  database  for  transparent  patient  data  collection 
and  analysis.  The  virtual  reality  exercise  consists  of  a  3-D 
piloting  of  an  airplane  through  multiple  hoops,  as  shown 
in  Figure  2  [4J. 

The  movements  allowed  by  the  platform,  pitch,  roll 
and  yaw,  resemble  the  motions  of  the  talo-crural  and 
subtalar  joints,  plantar  and  dorsiflexion,  inversion  and 
eversion  and  pronation  and  supination  [7],  The  number 
and  placement  of  the  hoops,  as  well  as  airplane  speed, 
viewpoint  to  the  simulation  and  platform  maximum  force 
levels  can  be  modified  by  the  therapist.  This  gives  the 
therapist  the  ability  to  adjust  the  level  of  difficulty  for 
each  exercise,  on  a  patient-specific  basis.  Adjustments  are 


based  on  baseline  performance  on  the  RA  and  exercise 
performance  during  a  training  session.  While  the  patient  is 
exercising,  force  and  displacement  data  are  displayed  in 
real  time  on  the  screen  and  compared  with  baseline 
information.  The  data  are  transparently  and 
simultaneously  stored  in  a  database,  which  then  calculates 
total  exercise  mechanical  effort,  peak  and  average 
forces/torques,  number  of  hoops  correctly  passed,  and 
other  variables.  Thus  the  healing  process  can  be  quantified 
based  on  objective  measures,  either  at  the  physical  level 
(range,  torque,  mechanical  effort),  or  functional  level 
(hoops  entered/missed,  number  of  hoops,  exercise 
time/level  of  difficulty). 


Figure  2.  Ankle  rehabilitation  exercise  in  VR  [4] 

©  Rutgers  University.  Reprinted  by  permission 

4.  Clinical  tests 

In  Summer  2000  the  Rutgers  Ankle  system  was 
installed  in  an  outpatient  rehabilitation  clinic  in  the  State 
of  New  Jersey,  for  a  two-week  trial  period.  The  goals  of 
the  pilot  tests  were  to  determine  how  well  the  current 
design  of  the  interface  worked  in  a  clinical  setting  and  to 
provide  preliminary  evidence  of  efficacy  (whether  patients 
benefited  from  incorporating  the  RA  into  their 
rehabilitation  program). 

4.1.  Subjects 

Patients  who  participated  in  the  two-week  trials  used 
the  Rutgers  Ankle  to  complement  their  regular 
rehabilitation  program.  This  paper  reports  on  three  of  the 
patients  who  ranged  in  age  from  14-56  and  who  had 
varying  musculoskeletal  ankle  injuries,  time  since  injury 


12 


Eurohaptics  2001  Birmingham  July  2001 


and  familiarity  with  computer  games.  One  of  the  subjects 
participating  in  the  study  is  shown  in  Figure  3. 


Figure  3.  Subject  participating  in  the  Rutgers  Ankle 
pilot  study.  ©  Rutgers  University.  Reprinted  by 


permission. 

4.2.  Method 

On  the  first  day  of  the  trial  the  subjects  received  a 
clinical  exam,  participated  in  baseline  testing  on  the  RA, 
were  instructed  and  practiced  on  the  RA  and  completed  a 
subjective  questionnaire.  The  clinical  exam  included  ankle 
strength,  range  of  motion  (ROM),  balance  and  functional 
ability  measures. 

Baseline  testing  on  the  Rutgers  Ankle  was  done  to 
measure  displacement  and  torque  for  all  ankle  motions. 
Subjective  evaluations  consisted  of  questions  in  which 
participants  were  asked  to  rate  their  experience  learning 
and  using  the  interface.  The  participants  exercised  on  the 
virtual  reality  system  for  five  sessions,  which  lasted 
approximately  30  minutes  each  and  were  scheduled  every 
other  weekday.  The  progression  of  training  was 
customized  for  each  patient  and  was  based  on  their 
performance  with  adjustments  made  for  intensity,  duration 
and  complexity  of  ankle  movements.  Baseline 
performance  was  obtained  for  each  session  in  order  to 
update  the  settings  of  the  Rutgers  Ankle. 

Training  was  supervised  by  a  physical  therapist  and 
was  a  complement  to  the  rehabilitation  program  for  two  of 
the  three  patients.  To  ensure  consistency  between 
sessions,  subjects  were  positioned  in  the  same  way,  their 
knee  flexion  held  within  five  degrees  of  flexion,  which 
was  measured  with  a  goniometer.  The  platform  was  held 
in  the  zero  position  prior  to  beginning  the  exercise.  On  the 
sixth  session  patients  underwent  a  clinical  exam,  re-testing 
on  the  Rutgers  Ankle  and  once  again  completed  a 
subjective  questionnaire.  The  three  subjects  completed  the 
six  exercise  sessions. 


4.3.  CASE  1 

The  patient  was  a  1 4-year  old  male  who  was  diagnosed 
with  a  Grade  I  inversion  sprain  that  occurred  during 
wrestling.  He  had  injured  his  ankle  two-weeks  prior  to 
beginning  the  trials.  His  goal  was  to  return  to  wrestling. 
He  had  extensive  exposure  to  computer  games,  which  he 
used  for  recreation.  The  subject  quickly  mastered  the 
basic  virtual  reality  simulation,  which  allowed  the 
investigators  to  test  the  upper  limits  of  the  system. 
Training  sessions  ranged  between  12  and  20  minutes. 
Simulations  were  executed  on  average  for  two  minutes. 
His  longest  continuous  session  was  performed  for  the 
combination  movement  at  a  speed  of  0.72  hoops/sec  and 
difficulty  setting  of  “medium”  for  ten  minutes.  He  quickly 
progressed  to  training  at  the  “difficult”  setting  and  the 
highest  level  of  resistance.  During  the  final  session  he  was 
able  to  increase  from  his  average  speed  of  0.72  to  1.2 
hoops/second.  The  subject  reported  enjoying  the 
simulation,  but  we  anticipated  that  he  would  lose  interest 
once  he  mastered  it.  He  did  not  appear  concerned  about 
making  errors  and  learned  to  anticipate  the  presentation  of 
the  targets.  Accuracy  for  the  simulation  improved  from 
the  first  to  the  final  session  from,  36-42%  for  pitch,  37% 
for  roll  and  38%  for  the  combination  movements  to  100% 
for  all  movements  at  a  comparable  speed,  level  of 
difficulty  and  resistance.  Figure  4  shows  the  increase  in 
task  accuracy  (%  of  hoops  entered)  over  all  simulations 
modalities. 


1  2  3  4  5  6 

Day 

Figure  4.  Subject  1  task  accuracy  (%  loops  entered) 
averaged  over  all  modalities  (ankle  pitch,  roll  and 
combination)  over  the  duration  of  therapy. 

Clinically  this  patient  presented  with  minor  deficits  in 
strength  and  balance  and  had  some  complaints  of  pain 
while  walking  down  the  stairs.  After  training  he  improved 
on  the  time  he  could  maintain  his  balance  on  his  right  leg 
with  his  eyes  closed  from  12  to  24  seconds.  He  also 
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decreased  the  amount  of  time  it  took  him  to  ascend  and 
descend  12  steps  from  23  to  18  seconds.  After  training  he 
did  not  have  any  pain  while  climbing  the  stairs.  At  the 
completion  of  the  trail  he  was  discharged  from  physical 
therapy  and  returned  to  wrestling. 

4.4.  CASE  2 

The  patient  was  a  15-year  old  female  who  was 
diagnosed  with  a  Grade  II  inversion  sprain  that  occurred 
while  playing  softball  She  incurred  the  sprain  five  months 
before  the  beginning  of  the  trial.  She  used  the  computer 
for  school  but  did  not  play  any  video  games.  The  subject 
progressed  from  the  second  level  of  resistance  to  the  third 
level.  She  also  progressed  from  the  “easy"  level  to  the 
“medium”  level,  with  only  a  couple  of  attempts  at  the 
most  difficult  level.  Training  sessions  varied  between  nine 
minutes  to  as  much  as  24  minutes.  A  typical  exercise  span 
was  two  minutes  long  with  her  longest  continuous  bout  at 
eight  minutes.  Figure  5  shows  her  affected  ankle 
plantarflexion  angles  at  the  stall  and  end  of  therapy. 


Affected  ankle  -  Plantar  Flexion  -  Peak  values 


Force  Level 


Figure  5.  Peak  values  for  plantar  flexion  angle  of  Subject 
2  affected  ankle,  over  the  duration  of  the  rehabilitation 
intervention.  Force  levels  1,2, 3, 4,  indicate  increasing 
platform  resistance  settings. 

The  training  aggravated  her  symptoms  and  her  ability 
to  work  on  the  system  was  influenced  by  her  pain 
complaints.  Therefore  the  power  output  fluctuated  front  a 
low  of  390  watts  to  a  high  of  2020  watts.  Often  after  the 
VR  simulation  ice  packs  were  applied  to  the  ankle.  Site 
reported  enjoying  the  simulation  initially,  but  after  many 
repetitions  she  revealed  that  she  found  it  somewhat 
boring.  At  the  end  of  training  her  clinical  exam  revealed 
improvements  in  strength  and  balance.  Strength  increased 
by  1  manual  muscle  test  grade  from  a  4  to  a  5  for  ankle 
dorsiflexion,  inversion  and  eversion.  She  was  able  to  stand 
on  her  involved  leg  for  60  seconds  compared  to  45  before 
training  and  was  able  to  stand  on  her  involved  leg  with  her 
eyes  closed  lor  28  seconds  compared  to  7  seconds  before 
training.  Recordings  from  the  Rutgers  Ankle  showed  that 


she  improved  in  her  anlde  dorsiflexion  plantarflexion  (see 
Figure  5),  dorsiflexor  and  everter  peak  ROM,  as  well  as 
her  dorsiflexor  and  everter  and  plantarflexor  torques  (see 
Figure  6).  Her  accuracy  with  the  VR,  simulation  from  the 
second  day  to  the  final  day  of  training  improved  from 
56%  for  pitch,  from  67%  for  roll  and  from  65%  for  the 
combination  movement  to  100%  for  all  the  movements. 
Currently  she  is  playing  softball. 


12  3  4 


Force  Level 

Figure  6.  Peak  values  for  dorsiflexion  torque  for  the 

affected  ankle  of  Subject  2,  over  the  duration  of  the 
rehabilitation  intervention.  Force  levels  1 ,2,3,4,  indicate 
increasing  platform  resistance  settings. 

4.5.  CASE  3 

The  third  patient  was  a  56-year  old  female  who  had  a 
bi-malleolar  fracture  The  fracture  had  been  reduced  with 
an  external  fixator  two  months  earlier.  Her  goal  was  to 
resume  recreational  hiking.  She  used  the  computer  in  her 
job  as  a  school  administrator  but  had  no  experience  with 
VR  games.  This  patient  required  that  some  of  her  motions 
be  held  constant.  For  example,  when  she  worked  on  the 
pitch  motion,  the  therapist  had  to  control  the  ankle  yaw  to 
avoid  having  the  foot  deviate  out  of  position. 

Early  in  the  training  the  therapist  had  to  manually 
guide  the  platform  and  provide  verbal  cues  to  assist  the 
patient  with  the  simulation.  She  performed  at  the  very  low 
speeds  of  0.04  and  0.07  hoops  per  second  on  the  first  day 
on  the  easy  ROM  setting.  After  the  simulation  was 
modified  to  better  represent  the  inversion/eversion 
movements  she  immediately  increased  her  speed  to  0.18 
hoops/second.  By  the  end  of  the  training  she  was  able  to 
exercise  at  0.6  hoops  /second  for  all  motions  at  the 
medium  ROM  setting.  Her  training  sessions  were  between 
15  and  22  minutes  with  her  longest  bout  for  a  single 
exercise  was  six  minutes.  She  reported  enjoying  the 
simulation  and  felt  it  was  a  good  addition  to  her  program. 

Clinically  she  demonstrated  improvements  in  ROM  (5 
degree  increase  in  plantarflexion),  balance  (increased 
single  limb  support  from  3  to  8  seconds),  strength  (1/2 
grade  increases  for  inversion  and  eversion)  and  speed 
(50%  decrease)  at  which  she  climbed  the  stairs.  Results 
from  the  Rutgers  Ankle  showed  that  she  made  gains  in 
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ROM  of  the  ankle  everters  and  plantarflexors,  which 
corresponded  with  clinical  measurements.  Her  ROM 
increased  for  plantarflexion  (50%)  and  eversion  (55%). 
The  amount  of  torque  she  could  generate  increased  for 
ankle  everters  (60%),  and  dorsiflexion  (25%).  At  the  end 
of  training  her  power  output  (calculated  from  the  variables 
collected  by  the  RA)  increased  from  175  watts  to  475 
watts  for  the  affected  ankle  and  surpassed  that  of  the 
unaffected  ankle.  This  is  illustrated  in  Figure  7.  Accuracy 
of  the  movements  increased  from  22%  for  pitch,  28%  for 
roll  and  26%  for  the  combined  movements  on  the  first  day 
to  1 00%  for  all  movements  on  the  sixth  day.  During  the 
training  period  she  returned  to  hiking. 


□  Affected  H  Unaffected 


Day 

Figure  7.  Peak  values  for  ankle  mechanical  power  for  the 
Subject  3,  over  the  duration  of  the  rehabilitation 
intervention. 

5.  Interpretation  /Discussion 

The  RA  was  successfully  introduced  into  a  clinical  setting. 
Patients  with  varying  diagnoses  and  ages  were  able  to 
learn  the  simulations  and  train  on  the  device.  The  greatest 
improvements  were  found  for  the  patient  with  bi-malleolar 
fracture  who  presented  with  hypomobility  and  strength 
deficits.  The  patients  with  the  sprains  were  functioning  at 
a  higher  level  and  would  have  benefited  from  having  their 
rehabilitation  done  in  standing  position.  Clinical 
improvements  were  noted  for  strength  and  range  of 
motion  as  well  as  for  balance  ability.  Clinical  exams  and 
VR-related  measures  showed  that  all  subjects  had  some 
degree  of  improvement  and  reported  that  they  were  able,  to 
use  the  device  and  enjoyed  it  as  a  complement  to  their 
regular  rehabilitation  program.  Only  one  subject  (case  2) 
was  not  receiving  concurrent  physical  therapy,  so  it  is 
difficult  to  conclusively  attribute  the  clinical  and 
functional  gains  to  the  VR  training  alone. 

All  three  subjects  were  able  to  complete  their  exercises 
sessions  without  any  prolonged  adverse  effects.  Although 
the  RA  was  installed  in  an  open  area  with  treatment  tables 


on  one  side  and  an  exercise  area  on  the  other,  the  three 
subjects  were  able  to  concentrate  on  the  VR  simulation. 
They  were  immersed  in  the  virtual  environment,  and  were 
not  distracted  by  other  patients  exercising  in  the  vicinity. 
The  mapping  of  the  foot  motions  to  the  RA  was  modified 
based  on  subjects  input  and  observed  difficulties  with 
selected  motions.  Specifically,  the  degree  of  yaw,  and  roll 
movement  that  accurately  corresponded  with  the  inversion 
and  eversion  motions  had  to  be  adjusted.  Difficulties  with 
using  the  system  were  encountered  with  maintaining  the 
pressure  in  the  pistons  during  longer  exercise  spans.  The 
compressor  often  became  overheated,  which  resulted  in  a 
drastic  reduction  in  its  output  pressure,  requiring  that 
exercise  bouts  be  interrupted.  In  addition,  the  seating 
system  was  cumbersome  when  making  adjustments  for 
patient  positioning,  taxing  the  therapist.  Finally,  the 
calibration  of  the  “zero”  starting  position  will  need  to  be 
refined. 

6.  Conclusion  and  Future  Work 


The  case  studies  described  here  constitute  the  first 
clinical  data  indicating  preliminary  evidence  of  efficacy  of 
the  Rutgers  Ankle  system  as  a  rehabilitation  device  for 
orthopedic  patients.  As  a  result  of  these  trials  several 
improvements  are  planned  to  the  RA  controller,  in  order 
to  increase  reliability,  reduce  vibrations  and  increase  force 
output  capabilities.  It  became  clear  that  pneumatic  control 
needs  accurate  temperature  compensation  of  the  pressure 
sensors,  otherwise  unwanted  vibrations  result.  The 
compressor  needs  to  have  sufficiently  large  storage  in 
order  to  avoid  frequent  starts  under  load,  which  in  turn 
result  in  overheating  and  the  subsequent,  inability  to 
maintain  high  pressures. 

Furthermore,  a  dual-platform  system  that  would  allow 
the  patient  to  exercise  while  standing  is  being  designed. 
This  will  allow  the  addition  of  virtual  exercises  in 
standing,  followed  by  more  clinical  trials.  In  addition, 
more  simulations  are  being  developed  with  the  goal  of 
providing  greater  variety  as  well  as  increasing  the 
ecological  validity  of  the  exercises.  By  providing  several 
kinds  of  VR  simulations  the  patient  boredom  may  be 
alleviated.  Studies  in  which  the  RA  replaces  selected 
aspects  of  rehabilitation  such  as  strength,  balance  and 
proprioceptive  training  will  assist  with  further  evaluating 
the  efficacy  of  the  RA. 
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Abstract 

This  paper  describes  an  initial  attempt  to  compare 
performance  levels  of  users  of  different  skill  levels  on  the 
Glasgow  Horse  Ovary  Palpation  Simulator  (HOPS). 
Experimental  participants  were  asked  to  identify  the 
position  and  size  of  follicles  on  the  surface  of  virtual  horse 
ovaries.  The  two  experimental  groups  were  made  up  of 
expert  and  novice  users.  Experienced  large  animal 
veterinarians  were  chosen  as  expert  users,  and  second 
year  veterinary  students  were  chosen  as  novice  users.  The 
initial  results  of  the  study  suggest  that  novice  users 
perform  better  than  expected.  Some  possible  reasons  for 
this  are  discussed. 

Keywords 

Haptic,  force  feedback,  medical  simulation,  virtual  reality 
training. 

Introduction 

Virtual  Reality  is  increasingly  being  recognised  as  a 
potential  tool  for  providing  training  in  medical  procedures. 
It  offers  a  safe  controllable,  environment  for  medical 
personnel  to  practice  and  learn  new  skills  with  no  risk  to 
patients.  However,  validation  of  a  medical  simulation 
proves  difficult.  Ethical  considerations  often  prevent 
doctors  who  are  trained  using  untried  methods  from 
working  on  patients.  Particularly  as  a  simulator  may 
provide  no  training,  or  may  even  have  a  negative  effect  on 
training.  Studies  have  shown  that  simulators  can  be  used 
to  improve  performance  on  the  simulator,  as  well  as 
psychomotor  skills  [8],  but  there  is  little  evidence  to 
suggest  that  these  improvements  carry  over  to  actual 
surgical  procedures.  O’Toole  et  al.  [10]  describe  an 
experiment  where  experienced  surgeons  perform 
significantly  better  than  medical  students  on  a  surgical 
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simulator.  They  conclude  that  their  simulator  may  be 
useful  in  quantifying  surgical  skill. 

Although  Virtual  Reality  simulation  is  a  relatively  young 
area  in  medical  training,  simulation  is  already  a  well 
established  method  of  providing  training  in  medicine. 
Students  gain  experience  in  certain  techniques  through  use 
of  plastic  or  rubber  models,  but  these  often  lack  realism 
and  provide  no  useful  feedback  to  the  trainee.  Surgical 
and  diagnostic  skills  can  also  be  improved  in  the  anatomy 
labs  that  are  incorporated  into  the  medicine  and  veterinary 
medicine  courses.  Again,  there  are  problems  since 
cadavers  are  a  scarce  resource,  and  are  not  generally 
reusable.  Living  tissue  can  also  have  noticeably  different 
haptic  properties  than  cadaver  tissue.  VR  medical 
simulators  have  the  potential  to  present  anatomical  and 
physiological  information  to  the  user  simultaneously  on 
reusable  models.  Simulations  currently  developed  can  be 
divided  into  those  that  provide  training  for  minimally 
invasive  surgery  (MIS),  surgery,  or  palpation  procedures. 
MIS  simulators  are  by  far  the  most  common.  In  a  MIS 
procedure,  surgeons  view  their  interaction  with  the  patient 
through  a  monitor,  and  hence,  it  lends  itself  to  a  virtual 
simulation.  The  Preop  endoscopic  simulator  [4]  developed 
by  HT  Medical  Systems  is  one  example  of  a  system 
combining  a  force  feedback  MIS  training  system  with 
anatomical  and  physiological  models.  Other  systems  exist 
to  simulate  other  MIS  procedures  such  as  arthroscopy  or 
laparoscopy.  SKATS  [1]  and  VE-KATS  [II]  present  knee 
arthroscopy  training  systems. 

Surgery  simulations  cover  a  wide  range  of  techniques 
using  different  surgical  instruments.  Cathsim  [2]  is  an 
example  of  a  commercially  available  training  system  for 
venipuncture.  Berkley  et  al.  [3]  present  a  simulation  for 
training  in  wound  suturing. 
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The  development  of  a  palpation  simulation  presents 
different  problems  than  development  of  a  surgery 
simulation.  During  surgery,  a  medical  practitioner 
interacts  with  the  patient  through  surgical  instruments,  so 
the  haptic  feedback  from  the  tissue  to  the  surgeon  is 
mediated  by  the  instruments.  In  palpation  procedures,  the 
doctor  is  in  direct  contact  with  the  patient.  Burdea  et  al.  [5] 
describe  one  of  the  few  examples  in  the  literature  of 
simulation  of  a  procedure  involving  palpation.  Their 
comparison  of  rubber  and  virtual  prostate  models  indicated 
that  although  rubber  models  provided  better  recognition, 
the  virtual  tumour  models  were  also  recognisable  by 
experienced  doctors.  Dinsmore  et  al.  [7]  also  describe  the 
development  of  a  palpation  simulator  for  training  in 
detection  of  sub  surface  tumours. 

This  report  describes  an  ongoing  experiment  that  uses  the 
Horse  Ovary  Palpation  Simulator  (HOPS)  developed  at 
Glasgow  University  to  compare  the  performance  of 
experienced  and  novice  veterinarians  in  horse  ovary 
palpation. 

Ovary  Palpation 

Traditionally,  students  are  taught  horse  ovary  palpation 
through  books,  lectures,  and  practical  experience. 
However,  the  high  cost  of  keeping  horses  often  leads  to  a 
large  ratio  of  students  to  horses.  As  ovary  palpation  is  a 
stressful  procedure  for  the  horse,  ethical  considerations 
limit  a  student’s  opportunity  to  gain  experience.  A  horse 
ovary  examination  can  be  difficult  for  a  veterinary  student 
to  perform,  but  can  also  be  fatal  to  the  horse  if  performed 
incorrectly.  Students  are  only  exposed  to  conditions  that 
occur  during  their  training,  and  may  not  get  experience  in 
diagnosing  rare  or  unusual  cases.  Virtual  Reality  offers  a 
method  of  providing  training  for  any  condition  that  has 
been  modelled.  Access  can  also  increased  for  seasonal 
examinations,  like  pregnancy  diagnosis,  as  the  simulator 
can  be  used  all  year  round. 

During  an  ovary  examination,  tire  veterinarian  inserts  a 
gloved  hand  into  the  pelvic  area  of  the  horse  through  the 
rectum.  The  veterinarian  must  search  through  the  pelvic 
region  of  the  horse  for  the  uterus.  The  ovaries  are  attached 
to  the  uterus,  and  each  can  be  found  by  following  either  the 
left  or  right  uterine  horn.  This  is  difficult  in  itself,  since 
die  veterinarian  must  perform  this  search  through  touch 
alone  while  wearing  gloves.  It  usually  requires  several 
attempts  before  an  inexperienced  student  can  locate  an 
ovary.  Once  located,  the  veterinarian  will  cup  the  ovary 
with  one  or  more  fingers,  and  palpate  it  using  his/her 
thumb.  In  particular  (s)he  will  look  for  any  abnormalities 
in  the  shape  or  surface  properties  of  the  ovary,  and  through 
training  and  experience,  will  be  able  diagnose  different 
conditions  through  touch  alone. 


For  the  purposes  of  HOPS,  follicles  of  different  sizes 
could  he  placed  on  the  ovary  models.  A  follicle  is  a 
spherical  fluid  filled  sac  that  grows  on  the  surface  of  an 
ovary  with  some  of  the  sac  existing  under  the  surface  of 
the  ovary.  It  will  typically  grow  from  very  small  -  a  few 
millimetres  -  to  a  few  centimetres  in  diameter.  As  the 
follicle  grows,  it  will  also  tend  to  move  towards  the  centre 
of  the  ovary.  Depending  on  the  size,  position  and  feel  of 
the  follicle  a  veterinarian  can  diagnose  the  stage  of 
ovulation  of  (he  horse.  There  may  be  many  follicles  on  an 
ovary,  but  only  one  active  follicle  exists  at  the  one  time. 

The  Glasgow  Horse  Ovary  Palpation  Simulator  (HOPS) 

HOPS  consists  of  a  left  and  right  ovary  model  fixed  in 
space.  The  two  skills  that  are  important  in  ovary  palpation 
are  locating  and  identifying  the  ovaries,  and  palpating  the 
ovaries.  HOPS  attempts  to  provide  training  to  veterinary 
students  in  the  palpation  stage  of  a  horse  ovary 
examination.  The  left  and  right  virtual  horse  ovaries  can 
be  seen  in  figure  1. 


Figure  1:  The  Horse  Ovary  Palpation  Simulator.  This 
environment  consists  of  a  left  and  right  ovary.  On  the 
bottom  half  of  the  left  ovary,  a  spherical  follicle  can  be 
seen.  The  user’s  cursor  is  shown  as  the  yellow  sphere  in 
the  centre. 

A  user  can  interact  with  HOPS  through  the  PHANToM 
force  feedback  device  from  Sensable  Technologies  [9]. 
The  haptic  properties  of  the  models  have  been  developed 
in  conjunction  with  experienced  horse  veterinarians  at 
Glasgow  University  Veterinary  School.  A  selection  of 
veterinarians  were  asked  to  set  softness,  friction  and 
damping  properties  for  the  models.  Using  this  method  a 
“good  approximation”  of  actual,  ovary  properties  was 
achieved. 

A  previous  study  using  HOPS  has  shown  that  over  one 
training  session,  participants  trained  using  the  HOPS 
simulator  perform  similarly  on  specimen  ovaries  than 
participants  trained  using  traditional  methods  [6], 
Participants  in  this  experiment  were  veterinary  students 
with  little  or  no  horse  rectalling  experience.  In  this  case, 
performance  was  based  on  the  correct  location  and  sizing 
of  a  single  follicle  on  the  virtual  ovaries.  This  study  also 
showed  that  there  was  a  low  percentage  of  correct 
identification  in  both  cases  (~11%  correct),  which  suggests 
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current  methods  can  be  improved  upon.  This  experiment 
will  build  on  the  previous  work  with  the  HOPS  simulator. 

Overview  of  Experiment 
Training 

The  experiment  was  split  into  training  and  task  stages.  As 
none  of  the  participants  had  any  previous  experience  in 
using  the  PHANToM,  they  were  initially  presented  with 
the  standard  ‘Blocks’  demo  developed  by  Sensable 
Technologies  to  familiarise  them  with  the  device.  The 
training  stage  focussed  on  training  users  to  locate  and 
distinguish  objects  by  size  and  softness  using  touch  alone. 
The  training  environment  consisted  of  two  spheres  (shown 
in  Figure  2). 


Figure  2:  The  training  environment  consisting  of  2 
spheres. 

In  the  initial  training  stage,  the  spheres  had  identical 
surface  properties  but  varied  in  size.  Participants  had  to 
locate  these  spheres  in  the  environment,  and  answer 
whether  the  left  or  right  sphere  was  larger  or  whether  they 
were  the  same  size.  Once  a  participant  answered,  (s)he 
was  presented  with  the  next  case.  Participants  had  to 
provide  five  correct  answers  before  moving  on  to  the  next 
training  stage. 

The  participants  were  next  presented  with  a  similar 
training  environment.  In  this  stage,  the  spheres  remained 
the  same  diameter,  but  the  softness  was  varied. 
Participants  were  asked  to  judge  which  sphere  was  softer. 
This  training  stage  was  completed  once  the  participant 
provided  five  correct  answers. 

Next  participants  were  introduced  to  the  HOPS 
environment.  They  were  shown  the  models,  then  allowed 
to  explore  them  through  touch  alone  for  five  minutes.  A 
small  follicle  was  present  on  the  front  of  the  bottom  left 
ovary.  The  training  stage  of  the  experiment  was  then 
complete. 

All  users  were  able  to  complete  the  training  stage.  The 
time  taken  in  completing  the  training  varied  between  18 
and  25  minutes. 

The  Task 

The  experimental  task  involved  identifying  follicles  on  the 
surface  of  the  virtual  ovaries  through  touch  alone. 
Participants  were  presented  with  the  same  eight  cases  hut 
in  a  random  order.  In  each  case,  zero,  one  or  more 


follicles  were  present  on  either  ovary  up  to  a  maximum  of 
five  follicles  in  total.  Each  participant  was  given  up  to  five 
minutes  to  explore  the  environment  while  identifying  all 
follicles.  Identification  of  a  follicle  involved  identifying 
its  position  -  either  left  or  right  ovary,  front  or  back  of  the 
ovary,  and  top  or  bottom  of  the  ovary  -  and  its  size. 
Participants  were  told  that  a  follicle  could  be  2cm,  3cm  or 
3.5cm  in  diameter.  Timing  information  for  each  case  was 
calculated  for  analysis.  Workload  measurements  were 
collected  from  all  participants  with  a  NASA  TLX 
workload  evaluation  form. 

There  were  two  subject  groups  involved  in  the  experiment. 

•  Group  A  consisted  of  second  year  veterinary  students 
from  Glasgow  University  Veterinary  School.  At  this 
stage  in  the  course,  students  have  some  knowledge  of 
horse  ovary  palpation  through  lectures,  but  have  no 
practical  ovary  palpation  experience. 

•  Group  B  consisted  of  experienced  large  animal 
veterinarians.  Each  participant  has  years  of 
experience  and  practice  in  large  animal  ovary 
palpation. 

In  this  initial  study,  group  A  contained  10  participants  and 
group  B  contained  3  participants. 

Experimental  Apparatus 

During  the  experiment,  users  interacted  with  the  virtual 
environments  using  a  PHANToM  1.0  with  the  standard 
thimble  attachment.  The  equipment  was  set  up  as  shown 
in  figure  3  such  that  a  participant  received  no  visual 
feedback.  Participants  also  wore  headphones  to  obscure 
noises  produced  by  the  PHANToM  motors. 


Figure  3:  Experimental  setup  during  haptic  training.  Tire 
screen  is  pointed  away  from  the  user  so  hc/she  receives 
no  graphical  feedback  to  feel  die  ovaries. 


Hypotheses 

1.  The  measured  performance  on  the  simulator  of 
experts  group  would  be  significantly  better  than  the 
performance  of  the  novice  group.  This  performance 
is  measured  in  accuracy  of  identifying  follicles. 

2.  The  expert  group  will  perform  examinations 
significantly  faster  than  the  novice  group. 
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3.  The  measured  workload  of  the  expert  group  A  would 
be  significantly  lower  than  the  workload  of  the 
novice  group.  The  expert  group  will  show  a 
significantly  higher  confidence  rating. 

Results 

Performance  on  the  Simulator 

Initial  results  suggest  that  hypothesis  1  will  not  be 
supported.  Of  the  22  follicles  present  in  the  experiment, 
both  groups  reported  identifying  similar  numbers  of 
follicles.  However  differences  are  noticed  in  the 
percentage  of  follicles  correctly  placed.  Group  A  placed 
70.9%  of  follicles  correctly,  where  as  group  B  placed 
40.9%  correctly.  The  expert  group  reported  problems  in 
placing  a  follicle  on  the  ovary  once  found.  In  a  real 
examination,  the  veterinarian  will  hold  the  follicle  while 
palpating,  and  will  therefore  have  an  idea  of  the  position  of 
any  follicle  found  with  respect  to  the  ovary  in  their  hand. 
This  is  not  the  case  in  the  virtual  model  since  users  are 
restricted  to  one  point  of  contact  with  the  environment. 


Number  found 
per  trial 

%  Correctly 
Positioned 

%  Correctly 
positioned  & 
sized 

Group  A 

18.8 

70.9% 

36.4% 

Group  B 

17 

40.9% 

13.6% 

Table  1:  Results  from  both  groups  of  performance  in 
identifying  follicles  on  the  ovary  surface. 


Differences  were  also  observed  in  the  techniques  used  by 
group  A  and  group  B  in  the  trials.  Participants  from  group 
A  tended  to  maintain  contact  with  the  ovary  being  search, 
and  maintain  a  steady  force  to  trace  the  shape  of  the  ovary. 
Participants  from  group  B  tended  to  move  across  the  ovary 
surface,  repeatedly  prodding  it  and  therefore  varying  force. 

Comparison  of  Timing  Data 

Figure  4  shows  the  comparison  of  timing  data  between  the 
two  groups.  There  are  suggestions  from  Cases  6,  7,  and  8 
that  the  experienced  veterinarians  may  complete  their 
examination  more  quickly,  however,  a  larger  expert  group 
is  required  before  any  significant  results  can  be  detected. 


Figure  4:  Comparison  of  the  average  time  to  complete  each 
examination 


Workload  Analysis 

Figure  5  show's  the  comparison  of  workload  between  the 
two  subject  groups. 


m  Group  A 
a  Group  B 


Figure  5:  Comparison  between  the  expert  and  novice 
groups  of  subjective  workload  for  the  task 

Although  the  expert  group  is  small,  it  is  important  to  note 
that  the  ‘Performance  Achieved’  and  the  ‘Confidence’ 
ratings  are  low.  This  is  possibly  down  to  the  fact  that 
locating  follicles  on  an  ovary  is  performed  differently  in 
the  simulator  and  in  the  real  examination.  Both  groups 
found  the  task  very  mentally  demanding. 

Future  Work 

The  next  stage  of  the  experiment  will  be  to  continue  with  a 
larger  number  of  expert  users.  The  expert  user  group  just 
now  is  too  small  to  be  able  to  draw  any  conclusions  about 
the  significance  of  results. 

During  the  experiment  cursor  path  and  user  force 
information  were  collected.  Further  analysis  will  be 
performed  on  this  data.  Initial  aims  will  be  to  analyse  the 
path  information  to  detect  any  differences  in  techniques 
between  the  user  groups,  and  look  for  similarities  within 
the  expert  group.  Information,  such  as  force  used  to 
palpate  an  ovary  will  be  examined. 

We  will  also  investigate  techniques  for  addressing  the 
problems  raised  by  the  expert  group  in  positioning  follicles 
on  an  ovary.  Initial  tests  will  involve  incorporating  two 
PHANToMs  into  the  workspace.  One  PHANToM  can 
therefore  be  used  to  palpate,  the  while  the  other  can 
control  the  movements  of  the  ovary.  This  would  provide  a 
means  of  reference  to  position  the  follicle  on  the  ovary. 
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Abstract 

Charpentier  (1891)  showed  that  when  objects  of  identical  mass  but  different  volume  are  lifted,  subjects 
consistently  report  that  the  smaller  object  is  heavier.  This  finding  is  preserved  even  when  subjects  are 
aware  that  the  objects  are  of  the  same  weight,  and  has  become  known  as  the  size-weight  illusion.  More 
recently,  Flanagan  &  Bandomir  (2000)  found  that  a  similar  effect  is  seen  when  subjects  pick  up 
weights  with  different  grasp  spans.  In  this  case,  weights  picked  up  with  a  narrow  grip  are  judged  to  be 
heavier  than  identical  weights  picked  up  with  a  wide  grip.  The  experiment  described  here  is  an 
extension  of  Flanagan  &  Bandomir’s  work,  which  eliminates  visual  feedback.  We  find  that  the  grasp- 
span  weight  illusion  is  present  without  the  aid  of  vision. 


Introduction 

Flanagan  &  Bandomir  (2000) 
examined  the  effect  of  the  grasp  span 
used  to  lift  an  object  on  the  lifter’s 
perception  of  its  weight  (note  that 
throughout  we  refer  to  Flanagan  & 
Bandomir’s  Experiments  1  and  2). 
They  asked  subjects  to  pick  objects  of 
different  weights,  and  report  whether 
the  object  was  lighter  or  heavier  than  a 
reference  object.  The  part  of  the  object 
used  to  pick  it  up  was  rectangular,  so 
the  object  could  be  grasped  across  the 
narrow  axis  (3cm)  or  across  the  wide 
axis  (7cm).  In  addition,  the  subject 
could  move  such  that  the  face  of  the 
object  presented  to  the  subject  was  the 
wider  or  the  narrower  face. 

This  study  found  that  subjects 
perceived  the  weight  to  be  lighter 
when  lifted  with  a  wide  grip,  and 
heavier  with  a  narrower  grip 
(Experiment  1).  This  finding  holds 
whether  the  subject  views  the  object 
from  the  front  or  from  the  side 
(Experiment  2). 


Flanagan  &  Bandomir’s  study  did  not 
exclude  the  classic  size-weight  illusion 
as  their  subjects  could  see  the  objects 
that  they  were  lifting,  instead  the 
object  was  rotated  such  that  a  wider  or 
a  narrower  face  of  the  object  was 
visible  to  the  subject.  The  study 
presented  here  corrects  this,  and  tests 
the  presence  of  a  grasp-span  weight 
illusion  without  vision. 


Method 

We  recruited  20  able-bodied  subjects, 
who  did  not  report  any  difficulty  with 
using  their  preferred  hand  at  the  time 
of  the  experiment.  15  men  and  5 
women  were  tested,  between  the  ages 
of  18  and  23  years.  Subjects  were  not 
rewarded  for  their  participation. 

Following  Flanagan  &  Bandomir  we 
used  a  forced-choice  paradigm,  in 
which  blindfolded  subjects  lifted  a  test 
weight,  selected  in  random  order  from 
weights  of  80g,  94g,  lOlg,  108g,  115g, 
123g,  130g,  137g  and  15  lg,  then  lifted 
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a  reference  weight  of  115g.  Subjects 
were  then  asked  to  report  whether  the 
test  weight  was  lighter  or  heavier  than 
the  reference.  Each  weight  could  be 
lifted  using  either  a  narrow  grip  or  a 
wide  grip,  giving  a  total  of  four 
conditions. 

A  standard  lifting  platfoimn  was  used 
for  each  weight,  with  a  central,  narrow 
grasping  site  (3.2cm)  and  an  outer, 
wider  grasp  site.  The  size  of  the  wide 
grip  could  be  varied  according  to  the 
size  of  the  subject’s  hand,  and  was 
either  9.6cm  or  11.35cm.  We  chose 
the  grip  size  such  that  the  subject  was 
just  able  to  pick  up  the  lifting  platform. 
The  two  groups  performed  statistically 
identically  (data  not  shown),  and  will 
be  treated  as  one  group.  The  grasp 
sites  had  strips  of  sandpaper  attached 
to  provide  a  high-friction  surface,  and 
to  reduce  the  grip  force  needed  to  lift 
the  object  (Flanagan  &  Wing  1997). 

The  subjects  were  instructed  to  lift  the 
platform  using  the  thumb  and  the  ring 
and  middle  fingers.  No  instructions 
were  given  as  to  how  to  lift  the  weight, 
but  all  subjects  were  observed  to  lift 
the  weight  by  about  10cm,  hold  it 
momentarily  then  place  it  down  again. 

Analysis 

The  responses  for  20  subjects  were 
pooled,  and  for  each  test  weight  in 
each  condition  we  found  the  proportion 
of  trials  where  the  test  weight  is 
reported  as  being  heavier  than  the 
reference  weight.  Each  subject  lifted 
each  weight  twice  in  each  condition,  so 
there  are  40  estimates  per  point  in  each 


condition.  The  point  of  subjective 
equality  (where  the  two  weights  are 
judged  to  be  identical)  was  found  for 
each  condition  by  fitting  a  sigmoid 
function  to  the  observed  data. 

We  performed  F-tests  on  pairs  of 
curves  to  determine  whether  or  not  the 
underlying  populations  were 
significantly  different. 

Results 

We  find  that  when  both  weights  are 
lifted  with  the  same  grip  width 
(narrow-narrow  or  wide-wide),  the 
curves  fitted  to  the  response  data  are 
not  significantly  different  to  each  other 
[F(2,  14)  =  1.16,  n.s.],  and  the  halfway 
points  of  these  curves  are  close  to  the 
true  value  of  the  reference  weight. 

However  when  the  weights  were  lifted 
with  different  grip  sizes  (wide-narrow 
or  narrow-wide),  subjects  report  that 
the  test  weight  is  heavier  when  it  is 
lifted  with  a  narrow  grip,  and  lighter 
when  lifted  with  a  wide  grip.  The 
fitted  curves  are  significantly  different 
[F(2,  14)  =  347.76,  p<0.001],  with  the 
curve  representing  the  narrow 
condition  shifted  to  the  left,  and  the 
wide  condition  to  the  right. 

Figure  1  shows  the  proportion  of 
subjects  who  reported  that  the  test 
weight  was  heavier  than  the  reference, 
for  each  weight  in  each  condition.  It  is 
clear  from  the  data  and  from  the  fitted 
sigmoids  that  the  subjects  were  more 
accurate  when  the  weights  were  lifted 
with  the  same  grasp  size  than  with 
different  grasp  sizes. 
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Figure  1.  These  figures  show  the  proportion  of  subjects  who  reported  that  the  test  weight  was  heavier 
than  the  reference  weight,  for  a  given  test  weight.  Figure  1A  shows  the  conditions  where  the  two 
weights  were  lifted  with  the  same  grasp  span  (W  =  wide,  N  =  narrow),  with  the  fitted  sigmoid  shown. 
IB  shows  the  conditions  where  the  grasp  spans  were  different  (NW  -  test  weight  narrow  grasp, 
reference  weight  wide;  WN  =  test  weight  wide,  reference  weight  narrow). 
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The  midpoints  of  the  fitted  sigmoids 
away  from  the  true  value  of  the 
reference  weight  give  the  points  of 
subjective  equality  for  the  different 
conditions.  In  the  condition  where  the 
test  weight  was  lifted  with  a  narrow 
grip  and  the  reference  weight  with  a 
wide  grip  (narrow-wide  condition),  the 
midpoint  is  shifted  to  the  left  by  11. 5g, 
suggesting  that  for  a  given  weight,  the 
perceived  weight  was  11. 5g  heavier 
than  the  true  weight.  For  the  wide- 
narrow  condition  it  is  shifted  to  the 
right  by  20.9g,  suggesting  that  it  is 
perceived  as  being  that  amount  lighter. 

Discussion 

The  experiment  described  here 
demonstrates  that  the  grasp-span 
weight  illusion  has  a  purely  haptic  or 
kinaesthetic  cause,  since  we  eliminated 
the  ability  of  subjects  to  use  vision  to 
pre-judge  the  propeities  of  the  object 
to  be  lifted. 

Since  the  only  parameter  of  the 
experiment  that  is  being  manipulated 
between  conditions  is  the  grasp  span, 
the  illusion  must  arise  either  from 
perception  of  the  grasp  size,  or  from 
the  perception  of  the  work  required 
from  the  hand  muscles. 

Van  Doren  (1998)  found  that  as  grasp 
span  increases,  the  amount  of  work 
required  of  the  hand  muscles 


decreases,  for  a  given  grip  force.  Our 
grip  spans  were  larger  than  those  that 
Van  Doren  measured  (up  to  8cm).  We 
believe  that  the  trend  should  hold  for 
larger  grip  spans,  however  this  study 
did  not  measure  muscle  activity  during 
the  task.  We  suggest  that  this 
differential  pattern  of  muscle  activity  is 
the  best  candidate  for  the  substrate  of 
the  grasp-span  weight  illusion. 
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Abstract 

Data  from  a  100-contactor  tactile  array  on  the  fingertip 
(1  mm  x  1  mm  matrix )  suggest  that  it  may  be  possible  to 
target  different  receptor  populations  in  the  skin  by  using 
different  frequencies  of  vibratory  stimulation  (40  Hz  and 
320  Hz).  Results  from  perception  of  a  moving  target 
within  a  uniform  "background"  vibration  indicate  that 
there  is  greater  spatial  acuity  for  perception  at  320  Hz 
than  at  40  Hz-  In  the  absence  of  a  background  vibration, 
discrimination  between  moving-bar  stimuli  presented  at 
resolutions  of  1  mm,  2  mm  and  3  mm  is  difficult, 
particularly  at  40  Hz,  suggesting  that  a  100-contactor 
tactile  array  may  offer  little  advantage  over  a  25- 
contactor  array  (2  mm  x  2  mm  matrix)  in  some  contexts. 


1.  Introduction 

In  "natural"  touch  perception,  sensations  are  produced 
by  spatiotemporal  patterns  of  mechanical  disturbance  at 
the  skin  surface.  At  Exeter  we  have  developed  a 
stimulator  array  which  is  designed  to  simulate  such 
sensations  on  the  fingertip  by  artificially  generating  the 
appropriate  patterns  of  tactile  stimulation.  The  spatial 
resolution  required  for  stimuli  is  determined  by  the 
density  of  touch  receptors  in  the  skin  -  around  1  mm'2  on 
the  fingertip.  To  achieve  this,  the  array  has  been  designed 
with  100  contactor  pins  arranged  on  a  1  mm  x  1  mm 
square  matrix  over  an  area  of  1  cm2  which  covers  the 
fingertip  (see  Figure  1).  Each  contactor  is  driven  by  a 
piezoelectric  actuator  (see  Figure  2)  -  the  100  drive 
waveforms  can  be  individually  specified  in  software  and 
are  delivered  via  a  purpose-built  interface. 

In  previous  tactile  arrays,  such  as  the  widely  used 
Optacon  device,  the  stimulus  waveform  from  each 
contactor  has  generally  been  limited  to  a  fixed  frequency 
(-250  Hz)  at  a  fixed  level,  with  stimulus  patterns 
produced  by  simple  on/off  keying.  The  Exeter  array  has  a 
significant  advantage  in  this  respect:  within  the  working 
bandwidth  of  25-400  Hz,  a  wide  variety  of  stimulus 
waveforms  is  possible  from  each  contactor.  This  allows, 
for  example,  targeting  of  different  receptor  systems  in  the 
skin,  which  differ  in  terms  of  their  frequency  response  [1]. 


Figure  1.  The  array  has  100  contactors  over  an  area  of 
1  cm2.  The  displacement  waveform  of  each  contactor 
can  be  individually  specified. 


Figure  2.  The  underside  of  the  array:  100  piezo-electric 
drivers  are  arranged  in  five  tiers. 
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2.  First  experiment 

This  is  an  investigation  of  the  perception  of  a  moving 
target  (created  by  contactor  pins  driven  at  amplitude  S ) 
against  a  uniform  background  (created  by  the  remainder 
of  the  array  driven  at  amplitude  N,  with  N  <S). 

2.1.  Method  and  results 

Subjects  are  required  to  identify  the  direction  of 
motion:  up,  down,  left  or  right.  The  target  is  in  the  form 
of  a  line  of  L  array  elements  (L  =  1,  3,  5,  7),  moving 
across  the  array  in  a  direction  perpendicular  to  the  line:  a 
horizontal  line  moves  up  or  down  and  a  vertical  line 
moves  left  or  right.  (For  L  =  1,  the  "dot"  moves  in  any  one 
of  the  four  directions.)  In  all  cases  the  target  moves  from 
one  side  of  the  array  to  the  other  in  one  second,  i.e.,  at  a 
speed  of  1  cm  per  second.  In  this  experiment,  all  the 
contactor  pins  (both  target  and  background)  are  driven 
with  sinewaves  at  the  same  frequency  /.  Half  of  the  test 
blocks  were  run  with/=  40  Hz  and  the  remainder  with 
f=  320  Hz.  These  frequencies  were  chosen  in  the  hope  of 
targeting  non-pacinian  and  pacinian  touch  receptors, 
respectively. 

With  the  target  amplitude  S  held  constant, 
discrimination  scores  were  determined  for  8  subjects  as  a 
function  of  the  background  amplitude  N.  Curve  fitting 
with  a  suitable  psychometric  function  was  used  for  each 
of  the  8  test  conditions  (4  values  of  L  with  two  values  of 
f)  to  obtain  in  each  case  an  estimate  of  the  critical 
background  amplitude  at  which  the  targets  are  just 
discriminable.  This  was  specified  as  the  value  of  N 
corresponding  to  a  discrimination  score  of  62.5  %  (i.e., 
halfway  down  the  psychometric  functions,  which  run 
from  100  %  to  the  chance  score  of  25  %).  Figure  3  shows 
these  values  of  critical  background  amplitude  (in  arbitrary 
units,  with  S  -  120  in  the  same  units). 


0  X - - - "1 - 1 - ■ 
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Figure  3.  Critical  values  of  background  amplitude  N  (at 
which  the  direction  of  movement  of  a  target  of 
amplitude  S  =  120  is  discriminable  with  62.5  % 
success  rate  [chance  =  25  %])  as  a  function  of  length 
of  the  target  line.  Data  are  shown  for  stimulus 
frequencies  of  40  Hz  and  320  Hz. 


At  320  Hz,  amplitude  100  corresponds  to  around  4 
microns  peak-to-peak,  hence  the  target  at  amplitude  120 
corresponds  to  around  5  microns  peak-to-peak;  at  40  Hz, 
amplitude  1 00  corresponds  to  around  40  microns  peak-to- 
peak,  hence  the  target  at  amplitude  120  corresponds  to 
around  50  microns  peak-to-peak.  The  conversion  between 
displacement  amplitude  and  sensation  level  is  not 
straightforward  -  the  detection  threshold  varies  with 
stimulation  area  (and  presumably  varies  differently  at  the 
two  stimulation  frequencies).  However,  the  bar  stimuli  are 
all  at  a  "comfortable"  sensation  level  and  the  1:10  ratio  of 
the  amplitudes  at  320  Hz  and  40  Hz  means  that 
corresponding  40  Hz  and  320  Hz  stimuli  are  at  roughly 
the  same  sensation  level. 

2.2.  Discussion 

Both  data  sets  flatten  at  a  line  length  of  5  mm  (i.e., 
L  =  5).  This  value  may  relate  to  the  size  of  the  receptive 
fields  involved  in  the  task.  There  is  a  significant 
difference  between  the  data  at  40  Hz  and  at  320  FIz.  This 
suggests  that  different  receptors  may  have  been  targeted 
with  the  two  different  stimulus  frequencies.  Certainly,  the 
sensations  are  very  different:  40  Hz  stimuli  produce  a 
much  better  approximation  to  "real"  touch  sensations  than 
do  320  Hz  stimuli.  However,  there  is  a  problem  with  this 
interpretation  in  terms  of  different  receptor  populations: 
there  are  suggestions  in  the  literature  that  non-pacinian 
receptors  have  greater  spatial  acuity  [2],  and  so  it  might 
be  expected  that  discrimination  in  this  "signal-in-noise" 
task  would  be  easier  at  40  Hz  than  at  320  Hz.  However 
the  experimental  data  suggest  that  the  converse  is  true: 
equivalent  performance  is  obtained  with  much  higher 
background  amplitudes  at  320  Hz  than  at  40  Hz, 
suggesting  that  spatial  acuity  is  greater  at  the  higher 
frequency. 

3.  Second  experiment 

Future  plans  include  construction  of  a  device  to 
address  four  fingers  and  thumb  on  one  hand,  each  digit 
with  a  5  x  5  array  of  2  mm  pitch.  It  is  thus  of  some 
interest  to  predict  the  difference  in  performance  between 
the  present  10  x  10  array  and  a  5  x  5  array,  each 
addressing  1  cm2  on  the  fingertip.  Here  we  describe  an 
experiment  on  perception  of  stimuli  at  different  spatial 
resolutions. 

3.1.  Method 

Test  stimuli  are  in  the  form  of  a  line  of  10  array 
elements  moving  across  the  array  at  a  speed  of  1  cm  per 
second  in  a  direction  perpendicular  to  the  line.  The  active 
contactor  pins  are  driven  with  sinewaves  at  frequency  /. 
Stimuli  move  across  the  array  via  9  increments  of  1  mm, 
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4  increments  of  2  mm,  3  increments  of  3  mm,  2  increments 
of  4  mm  or  1  increment  of  6  mm.  For  each  stimulus,  subjects 
are  required  to  identify  the  spatial  pattern  of  the  motion. 

Eight  volunteer  test  subjects  were  trained  and 
subsequently  tested  for  their  ability  to  distinguish  between 
stimuli  at  the  different  spatial  resolutions.  Each  subject 
was  presented  with  eight  test  blocks;  half  with  /  =  40  Hz 
and  half  with  /  =  320  Hz.  At  each  frequency  /,  separate 
test  blocks  were  run  for  each  direction  of  movement:  up, 
down,  left  or  right.  Each  block  consisted  of  20  test  items, 
distributed  between  the  5  different  spatial  resolutions. 
Noise  masking  via  headphones  was  used  to  eliminate  any 
acoustic  cues. 


3.2.  Results  and  discussion 
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Figure  4.  Representation  (not  to  scale)  of 
discrimination  index  between  the  five  stimulus 
resolutions.  (Data  pooled  over  stimulus  direction; 
upper  line:  40  Hz  stimuli;  lower  line:  320  Hz  stimuli.) 
The  labels  above  the  tic  marks  indicate  the  stimulus 
resolution  in  millimetres.  The  numbers  below  each 
line  indicate  the  calculated  discrimination  index  d  for 
neighbouring  stimulus  types.  At  40  Hz  the  d  between 
2  mm  and  3  mm  resolution  was  calculated  as  being 
very  slightly  negative  -  this  d  value  was  thus  forced 
to  zero.  A  second  set  of  data  at  40  Hz,  taken  in  a  pilot 
experiment  with  a  different  group  of  subjects,  is 
shown  in  brackets. 

Data  from  the  identification  task  were  obtained  in  the 
form  of  confusion  matrices,  indicating  the  extent  to  which 
the  five  spatial  resolutions  were  distinguishable.  No 
significant  differences  in  performance  were  observed 
between  the  four  directions  of  movement.  Hence  data 
were  pooled  over  direction  of  movement  and,  using  a 
method  similar  to  that  described  by  Braida  and  Durlach 
[3],  the  pooled  confusion-matrix  data  were  processed  to 
obtain  values  for  discrimination  index  d\  as  shown  in 


Figure  4.  (Note:  A  d’  value  of  less  than  1  indicates  that 
discrimination  is  difficult;  d'  values  are  cumulative,  e.g., 
the  d'  at  320  Hz  for  stimuli  at  2  mm  and  4  mm  resolution 
is  0.90+1.70  =  2.60.) 

These  data  show  poor  discrimination  between  stimuli 
at  1  mm,  2  mm  and  3  mm  resolution,  particularly  at 
40  Hz.  This  is  an  important  result,  since  it  suggests  that  a 
10  x  10  array  (1  mm  pitch)  offers  little  advantage  in  this 
context  over  a  5  x  5  array  (2  mm  pitch)  -  and  the  10  x  10 
array  has  significant  disadvantages  in  terms  of  cost,  control 
and  ease  of  construction. 

The  better  performance  for  320  Hz  stimuli,  compared 
to  that  for  40  Hz  stimuli,  can  be  related  to  the  fact  that 
discrete  "jumps"  in  the  moving  320  Hz  stimuli  are 
reported  to  be  clearly  detectable  at  all  resolutions, 
whereas  these  jumps  are  only  apparent  in  the  40  Hz 
stimuli  at  the  lower  resolutions. 

4.  Conclusion 

Each  experiment  provides  evidence,  albeit  indirectly, 
that  the  position  of  moving  stimuli  on  the  skin  is  more 
accurately  perceived  at  320  Hz  than  at  40  Hz.  It  is 
possible  that  this  relates  directly  to  the  nature  of  the 
stimulus  provided  by  the  array.  However,  the  design  of 
the  array  provides  no  obvious  mechanism  for  this,  and 
hence  we  believe  that  differences  between  results  at 
40  Hz  and  320  Hz  are  attributable  to  differences  in  the 
perceptual  mechanism  at  the  two  frequencies,  and  may  be 
explained  in  terms  of  the  involvement  of  different 
populations  of  receptors. 
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Abstract 

The  texture  of  a  real  or  virtual  surface  can  both  increase 
the  sense  of  realism  of  an  object  as  well  as  convey 
information  about  an  object’s  identity,  type,  location, 
function,  and  so  on.  It  is  crucial  therefore  that  interface 
designers  know  the  range  of  textural  information  available 
to  them  through  current  interaction  devices  in  virtual 
environments.  We  have  examined  roughness  perception  of 
a  set  of  force  feedback  generated  textures  ( conveyed  via  a 
PHANToM  device)  in  order  to  better  understand  the  range 
and  resolution  of  textural  information  available  through 
such  interaction.  We  propose  that  the  addition  of  audio 
stimuli  will  increase  further  the  potential  for  conveying 
more  varied  and  realistic  texture  percepts  through  force 
feedback  interaction.  We  are  currently  examining 
roughness  perception  of  a  set  of  auditoiy  stimuli  and  will 
use  both  sets  of  unimodal  results  to  test  the  potential 
benefits  of  combining  haptic  and  auditory  textural  stimuli. 

Keywords 

Haptic,  auditory,  force  feedback,  texture,  roughness 
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Introduction 

Despite  the  increasing  prevalence  of  haptics  in  today’s 
computing  environments,  the  effective  representation  of 
such  information  is  still  a  relatively  new  design  problem 
for  human  computer  interaction  research.  Force  feedback 
interfaces  in  particular  pose  a  variety  of  design  questions 
such  as  what  can  and  cannot  be  communicated 
convincingly  via  such  devices. 

In  human  sensing  and  manipulation  of  everyday 
objects,  the  perception  of  surface  texture  is  fundamental  to 
accurate  identification  of  an  object  [5].  In  a  virtual  world 
also,  texture  information  can  both  increase  the  sense  of 


realism  of  an  object  as  well  as  convey  information  about 
what  the  object  is,  where  it  is,  and  what  it  is  for  [4]. 
Through  force  feedback  interaction  in  particular  we  can 
provide  textural  information  that  we  can  literally  feel 
through  the  haptic  modality.  Given  that  it  is  often  argued 
that  touch  is  the  ’reality  sense’  [2],  being  able  to  feel  the 
texture  of  a  virtual  object  should  surely  lead  to  increased 
realism  of  the  object. 

Previous  work  investigating  the  perception  of  real 
surface  textures  has  shown  that  the  physical  properties  of 
textures  are  complex  and  that  an  overall  understanding  of 
texture  perception  remains  somewhat  elusive  [e.g.  3,4,5]. 
Textures  are  therefore  proving  difficult  to  reproduce 
successfully  in  virtual  environments.  It  has  been  accepted 
however  that  roughness  (along  with  hardness)  is  certainly 
one  of  the  primary  properties  of  a  surface  used  to  identify 
and  classify  an  object.  We  have  chosen  therefore  to  focus 
our  research  initially  on  this  dimension  of  roughness  of 
virtual  surfaces. 

Simulating  textures  with  force  feedback  devices  in 
particular  has  proved  an  interesting  research  problem. 
Force  feedback  devices  convey  texture  by  actuating 
kinesthetic  forces  on  the  users’ finger,  hand,  or  body.  This 
type  of  interaction  relies  on  forces  created  through 
kinesthetic  movement  or  displacement  of  the  device  and 
user  limbs  or  joints  while  much  of  the  texture  perception 
we  are  used  to  comes  through  tactile  stimulation  of  the 
mechanoreceptors  on  or  just  below  the  surface  of  our  skin 
[5].  We  have  found  in  our  previous  work  that  such  ’gross’ 
or  large  textures  can  perturb  the  users’ movements  so  much 
that  the  ability  to  stay  on  the  textured  surface  is  adversely 
affected  [7].  More  careful  design  of  such  force  feedback 
based  textures  is  required  if  these  devices  are  to  reach  their 
full  potential. 

High  fidelity  force  feedback  devices  (such  as  the 
PHANToM)  are  becoming  increasingly  realistic 
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interaction  tools  in  a  variety  of  applications  where  the 
texture  of  a  virtual  surface  may  be  of  great  importance. 
Medical  research  for  example  can  exploit  such  interaction 
in  surgical  and  diagnostic  simulations  where  the  texture  of 
tissue  or  organs  may  provide  crucial  information  or 
feedback  to  the  surgeon  during  a  procedure. 

Force  feedback  interaction  is  also  improving  the  ability 
to  design  and  prototype  a  variety  of  products  ranging  from 
the  commercial  (e.g.  cars)  to  artistic  and  historical  artifacts 
(e.g.  sculptures  and  jewelry).  E-Commerce  will  also 
benefit  in  that  companies  can  provide  their  customers  with 
a  close  representation  of  the  feel  of  their  products  before 
they  buy.  The  textile  and  fashion  industry  in  particular 
could  anticipate  increased  online  sales  if  the  texture  of  the 
clothing  could  be  felt  before  purchasing  f  1]. 

It  is  crucial  therefore  that  interaction  designers  know 
the  potential  range  of  textural  information  available 
through  each  modality  and  indeed  each  device  available  to 
them.  With  the  increasing  prevalence  of  force  feedback 
interaction,  it  is  particularly  important  to  establish  this  for 
the  haptic  modality  and  high  end  of  the  range  devices  such 
as  the  PIIANToM  (SensAble  Technologies). 

Past  research  suggests  that  texture  representation  is 
possible  through  force  feedback  interaction  but  that  the 
ideal  solution  is  yet  to  be  found.  This  is  due  in  part  to  the 
mismatch  between  real  texture  perception  (which  involves 
both  cutaneous  and  kinesthetic  sensation)  and  virtual 
texture  perception  (which  normally  relies  on  either 
cutaneous  sensation  through  tactile  devices  or  kinesthetic 
sensation  through  force  feedback  devices).  The  pioblem 
could  potentially  be  solved  by  advancing  the  currently 
available  devices  in  order  that  the  devices  better  suit  real 
texture  perception  [e.g.  8].  This  hardware-based  solution 
is  inevitable  as  the  technology  advances. 

Another  solution  may  be  to  improve  the  physical  and 
mathematical  modeling  of  real  textures  to  produce  the 
optimum  algorithms  for  generating  lealistic  virtual 
textures.  This  method  currently  has  mixed  results,  as  it 
cannot  be  assumed  that  the  virtual  exploration  of  texture 
matches  that  of  real  texture  perception.  Exact  physical 
modeling  therefore  may  be  pointless  if  the  interaction  used 
to  experience  the  texture  differs  significantly  from  that 
assumed  by  the  physical  model. 

Proposed  Solution 

Our  approach  offers  a  cost-effective  approach  that 
makes  use  of  the  currently  available  devices  and  even  the 
simplest  physical  models  of  texture.  We  propose  a 
multimodal  solution  that  exploits  the  human  ability  to 
combine  and  integrate  information  from  multiple  sensory 
modalities  into  a  fused  and  meaningful  and  whole  percept. 


We  hypothesise  that  presenting  combined  haptic  and  audio 
percepts  of  roughness  will  increase  the  quantity  and  quality 
of  textural  information  available  through  force  feedback 
interaction  alone. 

Overview  of  Experiments 

The  current  work  involved:  (1)  the  evaluation  of  the 
effect  of  texture  frequency  on  perceived  roughness  of  a  set 
of  force  feedback  generated  textures,  (2)  a  follow  up  study 
extending  the  range  of  frequencies  used  and  examining  the 
possibility  that  there  were  two  distinct  notions  of 
roughness  emerging  from  the  range  of  textural  stimuli 
used,  and  (3)  the  evaluation  of  the  effect  of  texture 
frequency  on  perceived  roughness  of  auditory  textures 
created  from  the  profiles  of  the  force  feedback  textures. 
This  perceptual  classification  process  will  serve  as  a  basis 
from  which  to  test  the  eventual  effects  of  systematically 
combining  the  haptic  and  auditory  texture  stimuli. 

The  Force  Feedback  Device 

The  PHANToM  1.0  force  feedback  device  by 
SensAble  Technologies  (Figure  1)  was  used  to  generate 
the  virtual  textures.  Optical  sensors  detect  changes  in  the 
device’s  configuration  and  mechanical  actuators  apply 
forces  back  to  the  user.  Users  interact  with  the  device  by 
holding  a  pen-like  stylus  attached  to  a  passive  gimbal  on 
the  device. 


Figure  1 :  The  PHANToM  3D  force  feedback  device  from 
SensAble  Technologies. 


By  scraping  this  stylus/probe  back  and  forth  across  the 
textured  area  the  appropriate  forces  can  be  calculated  from 
the  positional  information  of  the  tip  of  the  probe  and  the 
stored  algorithmic  models  of  the  textured  surface  with 
which  the  user  is  interacting. 
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Haptic  Textures 

Haptic  textures  were  generated  as  sinusoidal  waves  or 
gratings  on  a  rectangular  patch  on  the  back  wall  of  the 
workspace.  Figure  2  shows  a  diagrammatic  view  of  the 
profile  of  a  texture  and  the  forces  generated  as  a  result  of 
this  profile. 


(a)  (b) 


Figure  2;  (a)  diagrammatic  view  of  the  profile  of  the 
texture;  (b)  indication  of  forces  resulting  from  amplitude 
and  frequency  of  texture  wave. 

The  resulting  texture  profiles  depend  therefore  on  tire 
amplitude  and  frequency  of  the  sinusoidal  waves.  The 
textures  had  fixed  amplitude  of  0.5mm  and  variable 
frequency  (cycles  per  30mm).  The  frequencies  used  varied 
from  5  -  45.  Higher  frequencies  were  more  tightly  packed 
waves  and  lower  frequencies  were  more  loosely  packed 
waves.  The  result  of  these  textures  was  a  bump  felt  at  the 
peak  of  each  wave. 

Auditory  Textures 

Auditory  textures  were  generated  from  the  same 
sinusoidal  waves  on  a  rectangular  patch  on  the  back  wall 
of  the  workspace.  The  resulting  profile  still  depended  on 
the  amplitude  and  frequency  of  the  waves.  The  result  of 
these  textures  was  a  single  MIDI  note  generated  from  and 
heard  at  the  peak  of  each  wave.  No  experimental  forces 
were  experienced  through  the  device. 

Roughness  Comparisons 

Participants  in  Experiment  1  could  rate  the  textures  as 
the  same,  the  one  on  the  left  as  rougher,  or  the  one  on  the 
right  as  rougher.  Participants  in  Experiment  2  were  given 
the  same  options  but  with  the  additional  response  option  of 
rating  the  textures  as  not  comparable  on  the  same 
roughness  scale.  This  set  of  responses  allowed  us  to 
evaluate  (a)  whether  the  participant  perceived  the  two 
textures  as  the  same  or  as  different  in  terms  of  roughness, 
and  (b)  the  number  of  times  each  texture  was  rated  as  the 
roughest  of  the  pair. 

In  addition,  the  added  response  in  Experiment  2 
allowed  us  to  evaluate  (c)  which  textures  participants  felt 


were  different  but  not  comparable  along  the  same 
roughness  scale.  This  was  added  as  it  was  observed  in 
experiment  1  that  people  often  perceived  a  haptic 
difference  but  that  they  could  not  decide  easily  which  one 
was  in  fact  rougher. 

Procedure 

Participants  (Experiment  1,  N=12;  Experiment  2, 
N=10,  Experiment  3,  N=  12)  were  instructed  to  drag  the 
probe  of  the  device  over  each  of  the  indicated  textured 
surfaces  and  make  a  judgment  on  the  roughness  of  the  pair 
of  textures.  Participants  compared  each  texture  to  itself 
and  to  each  of  the  others  twice  (in  a  random  order).  In 
experiment  1,  subjects  compared  6  textures.  In  experiment 
2  the  frequency  range  was  extended  to  include  9  textures. 


Figure  3:  Interface  for  roughness  comparisons 


Participants  were  allowed  to  explore  each  of  the 
textures  during  that  trial  for  as  often  as  they  liked  and 
could  switch  between  exploring  the  one  of  the  left  to 
exploring  the  one  on  the  right  as  often  as  they  liked  to 
compare  the  two  textures.  They  were  instructed  however 
that  it  was  their  initial  response  to  the  textures  that 
mattered  most  and  that  there  were  not  necessarily  right  or 
wrong  answers  for  each  of  the  trials.  Participants  made 
their  response  by  clicking  the  switch  on  the  probe  of  the 
PHANToM  to  select  the  response  that  reflected  their 
roughness  judgment  for  each  trial. 

A  training  session  identical  to  the  experiment  but  with 
less  trials  allowed  the  participants  to  become  familiar  with 
the  device  and  the  interface.  Importantly,  it  also  allowed 
them  to  adopt  an  exploration  strategy  for  experiencing  the 
textures  comfortably  and  successfully. 

Hypotheses 

Independent  Variable:  frequency  of  texture  (cycles  per 
30mm). 

Dependent  Variable:  Perceived  roughness, 

operationalised  as  the  number  of  times  each  texture  was 
judged  as  the  roughest  of  the  pair. 
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Exp.  1:  The  frequency  of  the  haptic  texture  (or  number  of 
bumps)  will  have  an  effect  on  the  perceived  roughness  ot 
the  texture. 

Exp.  2-a:  Increasing  frequency  of  haptic  texture  (or 
number  of  bumps)  will  lead  to  an  increase  in  the  perceived 
roughness  of  the  texture. 

Exp.  2-b:  There  may  be  a  bimodal  function  of  roughness 
with  a  frequency  from  either  end  of  the  scale  being 
perceived  as  the  roughest  of  the  set. 

Exp.  2-c:  Textures  compared  from  either  end  of  the 
frequency  range  are  more  likely  to  be  rated  as  not 
comparable  on  the  same  roughness  scale  than  textures 
compared  within  the  high  range  or  within  the  low  range. 

Exp.  3:  The  effect  of  frequency  of  audio  texture  (or 
number  of  notes)  will  have  an  effect  on  the  perceived 
roughness  of  the  virtual  texture. 

Haptic  Results  (Experiment  1) 

Effects  of  Frequency  on  Perceived  Roughness 

The  frequency  of  the  texture  was  shown  to  have  a 
significant  effect  on  perceived  roughness.  That  is,  there 
was  a  significant  effect  of  frequency  on  the  number  of 
times  a  texture  was  judged  as  the  roughest  of  a  pair 
(F=9.73,  p<0.01).  The  number  of  times  each  (frequency 
of)  haptic  texture  was  judged  as  roughest  tended  to 
increase  as  the  frequency  of  the  texture  increased  (see 
Figure  4). 


Effect  of  texture  frequency  on  perceived 
roughness 


10  15  ZO  25  30  35 

Frequency  of  texture 
(cycles  per  30mm) 


Figure  4:  Effect  of  frequency  on  perceived  roughness. 

It  is  likely  however  that  the  range  used  in  the 
experiment  is  only  a  sample  from  a  more  complex 
function.  In  fact,  the  graph  shown  may  not  be  part  of  a 
simple  monotonically  increasing  function  at  all.  Instead  it 


may  be  part  of  a  quadratic  function  of  perceived  roughness 
as  suggested  by  people  such  as  Lederman  et  al.  [6].  At  the 
very  least,  it  may  be  likely  that  there  is  more  than  one 
maximum  roughness  generated  from  the  set  of  frequencies. 

Two  distinct  notions  of  haptic  virtual  roughness? 

Participant  comments  began  to  suggest  that  the  lower 
frequency  of  10  was  considered  very  rough  like 
corrugated  material’.  The  higher  frequencies  of  30  and  35 
however  were  also  labeled  as  very  rough  but  like 
sandpaper’.  It  is  possible  then  that  2  frequencies  from 
opposite  ends  of  the  scale  can  be  perceived  as  equal  in 
roughness  magnitude  but  from  different  roughness  scales. 

Experiment  2  extended  the  range  of  textures  (5-45 
cycles)  to  evaluate  whether  the  increasing  frequency 
leading  to  increasing  perceived  roughness  relationship  still 
held  beyond  the  range  used  in  Experiment  1  and  whether 
the  bimodal  peak  roughness  points  emerged.  This  follow 
up  study  also  evaluated  our  suggestion  from  Experiment  1 
that  comparing  two  textures  from  either  end  of  the 
frequency  range  would  increase  the  likelihood  that  they 
would  be  judged  as  different  but  also  increase  the 
likelihood  that  they  would  not  be  able  to  compare  the 
textures  on  the  same  roughness  scale.  Final  results  from 
this  evaluation  will  be  presented  at  the  workshop. 

Identical  Haptic  Stimuli 

Textures  with  equal  frequency  were  judged  as  the  same 
roughness  on  an  average  of  64%  of  the  trials.  It  appeared 
that  higher  identical  frequencies  were  more  likely  than 
lower  identical  frequencies  to  be  successfully  judged  as  the 
same.  This  could  perhaps  due  to  the  interaction  between 
probe  size  and  texture-profile  size  -  lower  frequencies 
being  more  susceptible  to  differences  in  hand  force  and 
exploration  speed.  Further  statistical  analysis  of  exp.l  and 
exp.2  will  investigate  this  hypothesis  further. 

Different  Haptic  Stimuli 

A  frequency  separation  of  5  cycles  was  not  sufficient  to 
significantly  separate  the  perceived  level  of  roughness  for 
the  haptic  textures  used.  That  is,  textures  separated  by  a 
frequency  difference  of  5  cycles  were  often  judged  as  the 
same  roughness.  As  frequency  differences  increased 
participants  found  it  increasingly  easy  to  decide  whether 
the  textures  felt  the  same  or  different  but  increasingly 
difficult  to  decide  which  of  the  two  was  in  fact  the 
roughest.  These  results  will  be  discussed  in  more  detail  at 
the  workshop. 
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Auditory  Roughness  (Experiment  3) 

The  audio  virtual  roughness  experiment  is  currently 
underway  and  the  effects  of  frequency  of  notes  on 
perceived  roughness  of  the  audio  textures  are  being 
evaluated  using  the  same  experimental  paradigm.  The 
MIDI  instrument  being  used  is  piano  although  this  will  be 
compared  to  other  instruments  in  the  future.  Our  main 
concern  for  the  initial  audio  experiment  was  purely  to 
explore  the  effects  of  frequency  of  an  arbitrary  sound  or 
note  on  the  perceived  roughness  of  the  auditory  texture. 
Results  from  the  auditory  roughness  experiment  will  also 
be  presented  at  the  workshop. 

Future  Work 

The  results  of  the  haptic  studies  suggest  that  larger 
frequency  differences  lead  to  more  easily  distinguishable 
textures  but  also  to  difficulties  in  using  the  dimension  of 
roughness  in  comparing  textures.  Large  textures  have  also 
been  found  to  throw  users  off  of  some  textured  areas  [7]. 
The  addition  of  audio  information  to  such  force  feedback 
textures  might  ameliorate  some  of  these  restrictions. 

We  propose  that  the  combined  (multisensory) 
presentation  of  haptic  and  audio  textural  information  will 
increase  the  range  and/or  resolution  of  textures  available  to 
the  designer  without  disturbing  interaction  through  force 
feedback  devices.  Results  from  the  unimodal  haptic  and 
audio  studies  will  be  presented  at  the  workshop.  Our  future 
multimodal  (haptic  -  audio)  experiment  will  also  be 
discussed  in  more  detail. 
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THE  ROLE  OF  FRICTION  AND  THE  RATE  OF  TANGENTIAL  FORCE  CHANGE 
IN  THE  SUBJECTIVE  SCALING  OF  ROUGHNESS. 

A.  M.  Smith,  C.  E.  Chapman,  M.  Deslandes,  J-S.  Langlais  and  M-Pierre  Thibodeau, 
Centre  de  recherche  en  sciences  neurologiques,  University  de  Montreal,  Montreal, 

Quebec,  Canada,  H3T  1J4 

Although  friction  was  originally  thought  to  be  an  important  parameter  in  the 
subjective  scaling  of  roughness  [1],  subsequent  experiments  by  Lederman  and  Taylor 
[4]  found  no  evidence  to  support  this  view  and  current  opinion  has  ruled  out  friction  as 
a  significant  determinant  [2],  However,  a  recent  study  by  Meftah  et  al  [3]  found  a  close 
comelation  between  the  distance  between  rows  of  surface  asperities  and  subjective 
roughness,  again  raising  the  question  about  whether  friction  co-varied  with  spatial 
period  and  roughness.  The  objective  of  the  present  study  was  to  examine  how  normal 
and  tangential  forces  at  the  fingertip  were  deployed  in  active  tactile  exploration  of 
textured  surfaces  in  order  to  determine  the  contribution  of  friction  to  the  subjective 
magnitude  estimations  of  roughness.  In  the  first  experiment,  six  volunteer  subjects 
were  asked  to  scale  the  roughness  of  8  surfaces  using  a  single  stroke  of  the  middle 
finger.  The  surfaces  were  7.5  cm  x  2.4-cm  polymer  strips  embossed  with  truncated 
cones  1.8  mm  high  with  a  spatial  period  of  2.0  mm  in  the  transverse  direction  and  1.5- 
8.5  mm  in  the  longitudinal,  scanning  direction.  The  surfaces  were  mounted  on  a  6-axis 
force  and  torque  sensor  that  measured  the  perpendicular,  contact  force,  normal  to  the 
skin  surface,  and  the  tangential  force  along  the  axis  of  stroking.  The  results  confirmed 
the  findings  of  an  earlier  study  that  magnitude  estimates  of  perceived  roughness 
increase  approximately  linearly  up  to  a  longitudinal  spatial  period  of  8.5  mm.  Despite 
stroking  speeds  that  ranged  from  10  mm/s  for  the  slowest  subject  to  157  mni/s  for 
fastest  subject,  there  was  no  discernable  relation  between  the  stroking  speeds  and  the 
roughness  estimates.  The  mean  tangential  forces  varied  from  0.42  to  1.28  N  and  the 
mean  normal  forces  varied  from  0.29  to  1.01  N.  However,  there  were  no  correlations 
that  were  consistent  for  all  subjects  between  perceived  roughness  and  either  the  mean 
tangential  or  normal  force  alone  for  each  spatial  period.  In  contrast,  the  correlations 
between  roughness  estimates  and  mean  friction  (the  ratio  of  the  tangential  to  normal 
force)  varied  between  0.65  and  0.88  for  the  six  subjects,  and  were  all  statistically 
significant  (p<0.01).  Examination  of  the  trial-by-trial  tangential  force  traces  revealed 
oscillations  whose  amplitude  increased  with  the  longitudinal  spatial  period  and  the 
frequency  was  determined  by  a  combination  of  the  spatial  period  and  the  stroking 
velocity.  These  oscillations  were  even  more  conspicuous  in  the  rate  of  change  of  the 
tangential  force  which  was  quantified  as  the  root  mean  square  (RMS)  of  the  tangential 
force  rate.  The  mean  normalized  RMS  proved  to  be  strongly  correlated  with  subjective 
roughness  averaging  0.88  for  all  subjects.  In  order  to  dissociate  the  fluctuations  in 
tangential  force  from  the  mean  kinetic  friction,  a  second  experiment  was  performed  on 
6  additional  subjects  who  estimated  the  roughness  of  identical  lubricated  and 
unlubricated  surfaces.  Lubrication  with  liquid  soap  respectively  reduced  the  mean 
kinetic  friction,  the  RMS  of  the  tangential  force  rate  and  also  the  subjective  estimates 
of  roughness.  Taken  together,  the  results  suggest  that  in  tactile  exploration,  the  RMS  of 
the  tangential  force  rate  may  be  an  important  determinant  of  subjective  roughness  and 
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a  significant  parameter  in  guiding  tactile  exploratioin.  According  to  these  results  a 
physiological  study  of  the  sensitivity  of  glabrous  skin  mechnaoreceptors  to  the  rate  of 
change  in  tangential  forces  would  seem  warrented.  Research  supported  by  the 
Canadian  Institutes  for  Health  Research. 
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Abstract 

Local  shape  is  an  important  attribute  that  can 
be  sensed  by  exploratory  movements  of  the 
finger.  Normally  this  involves  a  blend  of  tactile 
and  proprioceptive  cues.  A  curved  surface  will 
deform  the  pad  of  a  single  finger  and  so  tactile 
cues  can  indicate  whether  a  surface  is  convex 
or  concave  or  whether  it  slopes  one  way  or 
another.  When  more  subtle  discriminations 
must  be  made  between  different  degrees  of 
curvature,  scanning  motions  are  made  in 
which  the  finger  sweeps  along  the  surface.  In 
this  case  the  cue  to  cur\>ature  is  the  change  in 
position  of  the  finger  tip  over  time  and  here 
proprioceptive  input  is  important.  We  have 
been  examining  how  curvature  judgements  are 
affected  by  the  force  reflected  back  from  the 
curved  surface  during  scanning.  Normally 
when  you  run  your  finger  over  a  surface,  you 
experience  resistance  to  motion  due  to  friction. 
This  resistance  creates  a  force  vector  which 
varies  in  direction  with  friction.  But  the  vector 
also  varies  in  direction  with  the  curvature  of 
the  surface  traversed  by  the  finger.  We  used  a 
two-alternative  forced-choice  (2AFC)  task  in 
an  adaptive  staircase  in  which  subjects  made 
comparisons  between  various  test  curvatures 
and  a  reference  curvature  in  order  to  find  the 
point  of  subject  equality  (PSE)  between  the 
two.  Differences  in  friction  between  reference 
and  test  stimuli  were  found  to  alter  the  PSE  in 
a  consistent  manner.  In  particular,  we  found 
that  the  reference  curvature  was  only  closely 
matched  when  no  frictional  disparity  existed 
between  reference  and  test  surface.  Reference 
surfaces  that  exerted  high  frictional  forces 
produced  smaller  curvatures  as  PSEs  while 
surfaces  with  low  friction  produced  high 
curvature  matches.  These  results  suggest  that 
forces  experienced  in  palpating  a  surface  may 
be  utilised  in  the  comparison  of  curvature. 

1 .  Introduction. 

Curvature  is  an  important  local  shape 
descriptor  that  may  vary  continuously  across 
the  surface  of  an  object.  In  terms  of  the 
sensation  of  touch,  curvature  may  be  assessed 


by  either  the  cutaneous  receptors  of  the 
glabrous  skin  (finger  pads  or  palm)  of  the  hand 
or  through  proprioception  supported  by  several 
types  of  receptors  in  the  muscles  of  the  hand, 
or  both.  Evidence  for  the  former  type  of 
sensitivity  to  curvature  has  been  provided  by 
LaMotte  et  al  (1998);  see  also  Bisley  et  al 
(2000)  who  showed  that  slowly  adapting  type 
1  mechanoreceptors  in  the  monkey  hand 
respond  to  the  curvature  of  a  raised  bump  on 
an  otherwise  flat  surface.  Psychophysical 
experiments  have  also  shown  that  human 
subjects  are  very  good  at  judging  curvature 
using  only  the  finger  pads  (Goodwin  et  al, 
1991;  Goodwin  &  Wheat,  1992).  Evidence  for 
the  ability  to  perceive  curvature  through 
proprioception  has  been  studied  using 
psychophysical  experiments  on  humans.  Such 
experiments  involved  curvature  discrimination 
based  on  active  touch  or  passive  deformation 
of  the  whole  hand  (e.g.  Vogels  et  al,  1999; 
Pont  et  al,  1998).  Surface  curvature,  it  seems, 
may  be  assessed  equally  well  by  both  static 
placement  of  the  hand  across  the  surface  or  by 
active  touch  with  the  hand  or  fingers  (Pont  et 
al,  1999).  However,  it  has  been  argued 
elsewhere  that  active  touch  is  better  than 
passive  touch  in  the  discrimination  of  shape 
and  recognition  of  objects  because  the  former 
can  yield  continual  pick-up  of  information 
from  the  surface  (Gibson,  1966).  Differences 
in  the  discrimination  of  curvature  in  different 
parts  of  the  hand  were  also  studied  by  Pont  et 
at  (1997)  who  found  that  the  palmar  side  of  the 
hand  is  more  adept  at  this  task  than  the  dorsal 
side.  Pont  et  al.  attribute  this  to  the 
preponderance  of  cutaneous  input  from  the 
palm  although  the  task  clearly  involves  both 
(cutaneous  and  proprioceptive)  types  of  input. 

A  notable  feature  of  the  visual  sense  is  that 
of  maintaining  shape  invariance  or  shape 
constancy.  That  is,  a  shape  may  be  encoded  in 
geometrical  terms  equally  well  regardless  of 
which  visual  cues  are  being  used  to  depict  it. 
There  is  active  debate  in  the  vision  community 
whether  this  is  the  case  with  human  vision  and 
we  may  ask  whether  such  an  ability  is  found  in 
active  touch  as  well.  So  what  physical  factors 
might  influence  the  perception  of  shape  from 


36 


Eurohaptics  2001  Birmingham  July  2001 


visually  unaided  active  touch?  Imagine  that  a 
convex  bump  on  a  flat  surface  is  being  stroked 
by  a  single  finger  that  does  not  provide 
cutaneous  input.  Both  the  horizontal,  scanning 
and  vertical  velocity  of  the  finger  and  the 
reactive  force  of  the  surface  acting  on  the 
finger  may  be  affected  by  the  object  impeding 
motion  of  the  finger.  These  physical  variables 
may  be  encoded  by  the  mechanoreceptors  to 
the  muscles  of  the  upper  limb  and  therefore,  if 
curvature  can  be  encoded  by  through 
proprioception,  then  it  may  be  through  changes 
in  these  variables.  Surfaces  of  real  objects  also 
exert  an  additional,  frictional  force  on  the 
finger  in  a  direction  opposite  to  the  direction  of 
motion.  If  the  force  applied  through  the  finger 
is  greater  than  the  tangential  force  exerted  by 
friction  the  finger  starts  to  slide  across  the 
surface.  In  this  case,  dynamic  frictional  forces 
act  on  the  finger  to  impede  its  motion.  This 
resistive  force  depends  on  the  normal  force  at 
the  contact  surface  but  is  also  a  function  of  the 
coefficient  of  dynamic  friction  of  the  surface. 
It  is  therefore  possible  that  the  perception  of 
surface  curvature  is  influenced  by  the  frictional 
properties  of  the  surface. 

We  isolated  proprioceptive  cues  to 
curvature  by  using  the  Phantom  haptic 
interface.  The  Phantom  is  essentially  a  robot 
arm  controlled  by  three  motors  and  connected 
to  a  computer  that  can  control  the  exertion  of 
reactive  forces  to  a  single  finger  placed  in  a 
thimble.  These  forces  can  be  used  to  simulate 
solid  objects,  friction  and  viscosity.  The  useful 
feature  of  using  the  Phantom  is  that  sensory 
input  is  provided  by  active  palpation  without 
local  tactile  information  and  this  in  turn 
allowed  us  to  focus  on  the  contribution  of 
proprioceptive  cues. 

2.  Method. 

2.1  Simulation  of  Solid  Shape. 

The  Phantom  is  a  haptic  rendering  system 
that  uses  3D  force  feedback  to  generate  the 
impression  of  3D  solid  shape,  viscosity, 
friction  and  surface  texture.  The  haptic 
rendering  process  is  similar  to  that  in  computer 
graphics  (Salisbury  et  al.  1995;  see  also  Ho  et 
al,  1999).  For  instance  in  the  rendering  of  solid 
shape  the  position  of  the  cursor  or  stylus  is 
tracked  at  a  frequency  of  1  kHz  until  an 
intersection  with  a  virtual  surface  is  detected. 
Once  an  intersection  is  detected  various  forces 
can  be  applied  to  the  finger  to  give  not  only  the 
impression  of  solidity  but  also  of  surface 
compliance,  texture  and  surface  friction.  It 
should  be  noted  that  the  perception  of  solid 
shape  is  generated  only  as  a  function  of  the 


observers  active  movements  of  a  single  finger 
inserted  into  a  thimble.  If  there  is  no 
movement  (and  therefore  no  exerted  force) 
then  no  haptic  feedback  can  be  provided.  This 
allowed  us  to  isolate  the  proprioceptive  sense 
that  results  from  active  movement. 

As  well  as  generating  the  impression  of 
solid  shape  the  Phantom  can  also  be 
programmed  to  simulate  static  and  dynamic 
friction  (Salisbury  et  al.  1995).  Friction  is 
simulated  by  detecting  the  collision  between 
the  finger/stylus  and  an  object  in  the  scene  and 
applying  an  appropriate  tangential  force  to  the 
finger.  The  tangential  force  serves  to  restore 
the  finger  to  an  initial  position  as  the  finger 
attempts  to  slide  across  the  surface.  Thus  the 
finger  sticks  on  the  surface.  If  the  tangential 
force  required  to  restore  the  finger  becomes 
greater  than  the  normal  force  times  the 
coefficient  of  static  friction  then  sliding 
occurs.  During  sliding  a  tangential  force 
(proportional  to  the  coefficient  of  dynamic 
friction)  is  applied  to  the  finger  in  the  direction 
opposite  to  the  direction  of  motion,  which 
provides  the  sensation  of  dynamic  friction.  In 
this  manner,  the  surface  is  made  to  feel  either 
slippery  like  ice  (i.e.  low  friction)  and  feels 
sticky  like  rubber  (i.e.  high  friction). 

2.2  Stimulus  definition. 

The  curvature  (or  more  correctly  the  normal 
curvature)  of  a  surface  at  a  point  is  the 
curvature  of  a  section  of  the  surface  in  a  given 
direction  and  perpendicular  to  the  tangent 
plane  at  that  point.  On  a  sphere  the  normal 
curvature  is  the  same  in  all  directions.  On  a 
cylindrical  surface,  which  is  what  we  shall  be 
using  here,  the  normal  curvature  is  zero  along 
the  elongated  axis  and  maximal  in  a  direction 
perpendicular  to  this  axis.  Cross  sections  of  a 
right  circular  cylinder  are  circular  in  form  and 
the  maximal  normal  curvature  of  the  cylinder 
is  therefore  given  by  the  curvature  of  a  circular 
section  which  is  inversely  dependent  on  the 
radius. 

The  stimulus  in  this  experiment  was  a 
virtual  cylinder  that  the  subject  was  allowed  to 
stroke  twice.  This  virtual  surface  was 
generated  by  the  Phantom  force-feedback 
system.  This  was  implemented  using  the 
application  programming  interface  GHOST. 
The  Phantom  was  programmed  to  generate 
cylinders  with  variable  cross-section  and  it  was 
the  radius  of  these  cross  sections  that  served  as 
the  independent  variable  in  these  experiments. 
The  cylindrical  surface  was  defined  within  a 
virtual  space  measuring  5cmx20cmx20cm  (see 
figure)  with  the  X  axis  set  as  the  horizontal 
axis  perpendicular  to  the  desk  at  which  the 
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subject  sat.  The  Y  axis  was  the  gravitational 
vertical  axis  and  the  Z  (depth)  axis  defined  the 
forward-backward  direction  with  respect  to  the 
subject.  The  principal  axis  of  the  cylindrical 
segment  was  oriented  along  the  X  axis  and  the 
subject  stroked  the  segment  from  the  top  in  the 
forward-backward  (Z)  direction  .  Movement  in 
the  X  direction  was  restricted  so  that  strokes 
across  the  surface  followed  a  similar  trajectory 
each  time. 


Virtual  button 


Figure  1:  Schematic  representation  of  object 
surfaces  created  by  haptic  rendering. 


2.3  Task. 

The  aim  of  this  experiment  was  to 
determine  how  well  a  subject  could 
discriminate  between  the  curvature  of  two 
circular  strips  and  to  test  whether  this  ability  is 
affected  by  friction.  The  only  curvature 
information  was  through  proprioceptive 
sources  arising  from  the  Phantom.  We 
measured  the  point  of  subjective  equality 
(PSE)  for  two  curvatures  (0.013/cm  and 
0.008/cm).  The  curvature  was  manipulated  by 
varying  the  radius  of  the  cross  sections  of  a 
circular  cylinder  (i.e.  75mm  and  125mm) 
however  for  a  constant  position  of  the  centre  of 
curvature  this  manipulation  of  radius  would 
have  resulted  in  an  additional  height  cue  in  the 
y  direction  (see  Figure  1).  This  was  controlled 
by  allowing  the  centre  of  curvature  to  move  up 
or  down  in  the  Y  direction  so  that  the  maximal 
height  of  the  cylinder  remained  constant. 

The  subject  was  asked  to  determine  which 
of  two  (sequentially  presented)  curvatures  was 
greater.  One  of  the  two  stimuli  was  the 
constant  reference  (radius  —  75mm  or  125  mm) 
and  the  other  varied  according  to  a  1-up/l- 
down  adaptive  staircase.  After  having  stroked 
both  strips  the  subject  had  to  respond  using  a 
computer  keyboard  which  had  the  greater 
curvature.  Using  this  method  the  PSE  was 


estimated  as  the  average  of  six  reversals.  In 
order  to  bring  the  test  stimulus  quickly  to  the 
PSE  each  staircase  measurement  was  preceded 
by  4  ‘reducing  reversals’  in  which  the 
increment  or  decrement  of  the  test  was  varied 
as  a  function  of  the  reducing  reversal  number. 

The  influence  of  dynamic  friction  was 
tested  by  introducing  disparities  in  simulated 
friction  coefficient  between  the  reference  and 
test  stimuli.  Two  friction  coefficients  were 
used  (p-0.2  and  p=0.8)  for  both  the  reference 
and  test  stimuli.  The  p=0.2  stimuli  felt 
slippery,  rather  like  moving  ice  on  ice.  The 
p=0.8  stimuli  felt  rubbery.  In  both  cases  the 
friction  coefficients  were  low  enough  that 
smooth  and  uninterrupted  movement  of  the 
finger  across  the  strip  was  still  possible. 

2.4  Design. 

The  two  sizes  of  reference  stimulus  and  the 
two  friction  values  for  both  test  and  reference 
resulted  in  8  distinct  conditions  in  a  balanced 
design.  That  is,  each  condition  consisted  of  a 
unique  value  of  reference  radius  (either  75mm 
or  125mm),  reference  friction  (either  0.2  or 
0.8)  and  test  friction  (either  0.2  or  0.8).  Mean 
PSEs  were  collected  for  each  of  these  8 
conditions  in  random  order  and  repeated  5 
times  for  each  subject. 

2.5  Subjects. 

Four  subjects  participated  in  this  experiment 
and  were  rewarded  with  gift  vouchers.  All 
subjects  were  spent  several  minutes  training  on 
the  task  prior  to  the  actual  data  collection. 
Subjects’  view  of  their  hands  was  occluded 
during  the  experiment  and  they  were  asked  to 
close  their  eyes. 
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Figure  2:  Effects  of  reference  radius  and 
surface  friction  on  point  of  subjective  equality 
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3.  Results. 

The  dependent  measure  for  this  experiment 
was  the  PSE  radius  which  is  defined  as  the 
radius  of  the  test  surface  that  subjects  equate 
with  the  magnitude  of  a  given  reference  radius 
regardless  of  frictional  differences.  We  report 
results  mainly  in  terms  of  the  radius  of 
curvature  because  it  involves  more  intuitive 
and  manageable  dimensions  of  size  in  terms  of 
millimetres  rather  fractional  curvature 
quantities  although  it  should  be  remembered 
that  a  smaller  radius  of  curvature  produces  a 
larger  surface  curvature.  Figure  2  shows  the 
PSE  radius  averaged  across  all  4  subjects  and 
plotted  as  a  function  of  the  reference  radius. 
The  four  lines  correspond  to  the  four 
conditions  used.  The  reference  radius  was 
matched  closely  only  when  the  friction 
coefficients  for  test  and  reference  stimuli  was 
the  same  (i.e.  when  pr  =  pt).  When  there  were 
disparities  between  the  test  and  reference 
stimulus  friction  systematic  errors  in  mean 
PSE  were  observed.  For  example,  when  the 
reference  friction  coefficient  was  greater  than 
the  test  stimulus  friction  (i.e.  when  Pr>  p,)  the 
mean  PSE  radius  obtained  was  at  least  25% 
higher  than  the  actual  reference  radius  which 
meant  an  underestimation  of  the  radius  of 
curvature  of  the  reference  stimulus  (that  is  the 
reference  appeared  more  curved  than  it  really 
was).  For  trials  in  which  the  reference  surface 
friction  was  less  than  the  test  surface  friction 
(i.e.  when  gr  <  m)  the  mean  PSE  radius  was 
found  to  be  at  least  1 1%  lower  than  that  of  the 
reference  stimulus.  These  results  mean  that  the 
radius  of  curvature  of  the  reference  stimulus 
was  underestimated  for  high  friction  surfaces 
and  overestimated  for  low  friction  surfaces. 

An  analysis  of  variance  was  performed  on 
the  data  with  three  repeated  measures 
(reference  friction  pr,  test  friction  p,  and 
reference  radius)  and  mean  PSE  radius  as 
dependent  variable.  Assuming  an  a=0.05  as 
our  level  of  significance  we  found  a  significant 
main  effect  of  reference  friction  [Fu  =  1 1 .42, 
p<0.05],  an  almost  significant  effect  of  test 
stimulus  friction  [F1>3  =  11.42,  pK).09],  and  a 
highly  significant  main  effect  of  radius.  None 
of  the  interactions  (i.e.  between  pr  and  radius) 
was  significant. 

In  terms  of  repeatability,  we  use  the 
standard  deviation  of  settings  to  assess 
differences  in  relative  difficulty.  Standard 
deviations  were  calculated  for  the  five  repeated 
settings  made  by  subjects  for  each  of  the  8  data 
points.  The  mean  standard  deviations 
(averaged  across  all  subjects)  for  the  75mm 
and  125mm  circular  segments  were  1 1mm  and 
21mm  respectively.  A  paired  comparison 


students  t-test  showed  that  this  difference  in 
means  was  significant.  The  increased  variance 
of  settings  for  the  125mm  radius  strip  reflects 
greater  difficulty  in  assessing  a  shallower 
curvature  (larger  radius  of  curvature). 

4.  Conclusions. 

We  have  described  an  experiment  in  which 
subjects  made  curvature  comparisons  between 
sequentially  presented  curved  strips,  simulated 
using  the  Phantom  haptic  interface.  The  results 
show  clearly  that  simulated  curved  strips  can 
be  accurately  discriminated  using  kinesthetic 
information  derived  from  the  hand  joints  as  the 
finger  is  deflected  by  the  surface  during 
stroking.  Our  results  also  highlight  the 
importance  of  friction,  and  consequently  of 
reactive  forces,  on  the  curvature  discrimination 
process.  We  found  that  the  radius  of  strips  with 
a  higher  frictional  coefficient  than  the 
comparison  stimulus  was  overestimated.  The 
radius  of  strips  with  a  lower  frictional 
coefficient  than  the  reference  was 
underestimated.  In  terms  of  curvature,  this 
means  that  the  curvature  of  rough  or  high 
friction  surfaces  is  underestimated  relative  to 
low  friction  surfaces  and  our  results  show  that 
the  converse  of  this  is  also  true.  When  the 
friction  coefficients  of  the  reference  and  test 
stimuli  were  the  same  subjects  settings  of  the 
two  radii,  and  therefore  curvatures,  were  a 
close  match.  It  remains  for  us  to  explain 
exactly  why  disparities  in  friction  cause  under¬ 
and  over-estimates  in  curvature  in  these 
experiments  and  we  are  investigating  three 
potential  sources: 

1 )  The  subjects  use  tangential  forces  to 
encode  curvature  and  tangential  forces 
arising  from  friction  disrupt  this  encoding. 

2)  It  is  not  force  but  change  in  velocity  that  is 
used  to  encode  curvature.  The  frictional 
differences  disrupt  direct  velocity 
comparisons. 

3)  The  result  is  an  artifact  of  the  process  of 
generating  virtual  shapes  and  virtual 
frictional  forces. 
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Abstract 

The  spatial  acuity  of  the  skin  for  pressure  stimuli  has 
been  explored  extensively,  but  there  have  been  few 
attempts  to  study  spatial  resolution  for  vibratory  stimuli 
of  the  type  used  in  tactile  aids  for  sensory  disability  or  to 
augment  communication.  Because  vibrotactile  spatial 
acuity  has  never  been  empirically  determined  at  such 
sites,  pattern  perception  with  these  devices  might  be 
poorer  than  it  could  be  if  the  stimulus  parameters  that 
optimize  localization  were  employed.  Vibrotactile 
localization  was  measured  on  forearm  and  abdominal 
sites  used  by  wearable  tactile  communication  aids. 
Surprisingly,  stimulus  frequency  did  not  affect 
localization  on  the  arm,  tested  on  8  Os  for  a  given 
spatial  arrangement;  both  on  the  arm  and  trunk  (tested 
with  8  Os),  fewer  than  half  of  the  sites  were  uniquely 
identified.  Because  tactile  aids  should  be  usable 
throughout  a  person's  life  span,  especially  in  the  later 
years,  older  persons  will  also  be  tested  in  these  studies  of 
the  spatial  aspects  of  vibrotactile  pattern  perception. 

1.  Introduction 

Tactile  devices  to  augment  communication  are 
becoming  more  common  in  everyday  life,  especially  to 
serve  as  prostheses  for  sensory  disability  or.  Such 
systems  include  speechreading  aids  used  by  deaf  persons 
(e.g„  the  Tactaid,  worn  on  the  forearm  or  chest),  reading 
machines  for  blind  persons  (the  Optacon,  used  on  the 
fingertip),  spatial  orientation  systems  used  by  pilots  or 
astronauts  (the  Tactile  Situation  Awareness  System,  on  the 
trunk  of  the  body),  and  even  vibrating  pagers  and  cell 
phones.  In  every  case,  vibrotactile  spatial  acuity  has 
never  been  empirically  determined  at  the  application  site, 


so  pattern  perception  might  be  poorer  than  it  could  be  if 
the  parameters  that  might  optimize  localization  were 
known.  We  are  currently  exploring  localization  ability 
and  vibrotactile  spatial  acuity  at  a  number  of  these. 
Because  it  is  likely  that  stimulus  parameters  such  as 
vibration  frequency  might  affect  localization  at  a  site, 
these  are  also  being  varied.  The  two  sites  we  have 
examined  initially  are  the  trunk  of  the  body  and  the 
forearm.  In  this  paper  we  will  describe  localization  ability 
and  information  transfer  on  these  sites.  Because  tactile 
aids  should  be  usable  throughout  a  person’s  life  span, 
older  persons  are  also  being  tested  in  these  studies  of 
spatial  vibrotactile  pattern  perception.  The  long-term 
objective  of  this  research  program  is  to  understand  better 
both  the  sensory  and  perceptual  processes  that  underlie 
rapid  and  accurate  performance  with  cutaneous 
communication  systems. 

1.1.  History 

Spatial  acuity  has  been  used  as  a  measure  of  tactile 
sensitivity  for  over  150  years.  Weber’s  classic  1826  work 
[62]  describes  attempts  to  measure  tactile  acuity  on  a 
number  of  body  sites.  The  two  primary  methods  used 
then  (and  now-  e.g.,  [63])  are  to  measure  the  minimal 
separation  required  to  discriminate  one  versus  two  points 
touching  the  skin,  the  2-point  threshold,  and  error  of 
localization,  which  is  the  accuracy  with  which  tire  location 
of  a  touch  can  be  identified.  Over  the  years,  it  has 
become  clear  that  methodology  plays  a  large  role  in 
determining  the  threshold  levels  obtained  [6],  [18],  [40]. 
Johnson  [34]  and  Stevens  [53,  54]  describe  improvements 
in  the  methodology  and  the  devices  used  for  testing  that 
reduce  subjective  biases  and  more  appropriately  measure 
the  minimum  distance  that  can  be  perceived  between  two 
durative  points  touching  the  skin.  Nevertheless,  serious 
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problems  remain.  Tawney  [55]  and,  then  one  hundred 
years  later,  Johnson  [35],  argued  that  the  two-point 
threshold  cannot  measure  tactile  spatial  resolution  because 
spurious  cues  such  as  intensitive  differences  are  inherent 
in  the  presentation  of  single  versus  paired  points.  Even  a 
pin-prick  has  a  spatial  quality,  and  produces  a  sensation  of 
tactile  “extensity”  ([55]  p.  592).  The  traditionally- 
obtained  thresholds  therefore  may  only  represent  “the 
categorical  perception  of  a  distance  at  which  separation 
between  two  points  is  no  longer  ambiguous”  [64]. 

The  history  of  the  study  of  tactile  acuity,  rarely  has 
included  the  minimum  separation  on  the  skin  required  to 
resolve  vibratory  stimuli,  perhaps  because  it  is  so  difficult. 
Kirman  [37]  elaborated  on  a  number  of  demonstrations  of 
the  apparent  inability  to  resolve  simultaneous  vibrotactile 
stimuli,  as  did  Craig  [16],  Yet,  the  simple  question  - 
“How  close  together  can  we  place  vibrating  tactors  before 
their  loci  become  indistinguishable?”  -  is  still  often  raised 
by  designers  of  wearable  multi-tactor  arrays.  Typically 
the  two-point  limens  for  touch  are  cited  to  support  a 
particular  spatial  design  for  vibrotactile  arrays.  Yet,  one 
must  assume  that  vibratory  stimulation  will  extend  well 
beyond  the  limits  imposed  by  contactor  boundaries. 
Analyses  of  the  waves 


Figure  1.  Mechanical  wave  propagation  on  the  skin  of 
the  dorsal  thigh  produced  by  a  68  Hz  vibration  (From 
Franke  [21]). 


propagated  on  and  in  the  skin  from  vibratory  stimuli  like 
those  shown  in  Figure  1  by  Franke  [21]  and  Keidel  [36] 
suggest  a  number  of  parameters  that  can  influence  the 
skin’s  resolution  for  vibratory  stimuli.  A  static  surround 
may  damp  the  spread  of  surface  waves  [26],  but  not  those 
passing  through  deeper  tissues,  where  many  of  the 
cutaneous  receptors  lie,  as  shown  (for  hairy  skin)  in 
Figure  2.  The  velocity  of  traveling  waves  in  the  human 
skin  not  only  depends  on  the  viscoelastic 


Figure  2.  Cross-section  of  human  hairy  skin.  (From 
Greenspan  &  Bolanowski  [32]). 

properties  of  the  skin  itself,  but  also  on  factors  as  varied 
as  vibration  frequency,  skin  temperature,  and  whether 
underlying  tissue  is  bone,  fat,  muscle,  or  a  combination. 

Clues  to  the  relevant  variables  in  any  study  of 
vibrotactile  spatial  acuity  can  be  found  in  studies  in  which 
two  (or  more)  stimuli  were  used  to  examine  other  aspects 
of  tactile  processing.  Profound  interactions  are  revealed 
among  stimuli,  particularly  when  presented  close  together 
in  space  and  in  time  [37].  For  example,  B6k6sy  [2,  3] 
described  increases  and  decreases  in  apparent  location 
and  perceived  magnitude  of  the  stimuli  when  two 
vibrating  points  are  brought  close  together  spatially  and 
temporally  (within  cm  and  ms).  Geldard,  in  his  studies  of 
sensory  saltation  [24,  25]  demonstrated  a  number  of  ways 
in  which  the  apparent  locations  of  tactile,  visual,  and 
auditory  stimuli  shift  with  larger  temporal  disparities. 
Similarly,  Gilson  [29],  Sherrick  [51],  and  others  have 
examined  the  “masking”  interactions  that  occur  when  two 
vibratory  stimuli  are  presented  close  together  in  space 
and/or  time,  finding  that  the  closer  the  two  are  to  one 
another  (on  either  dimension),  the  more  the  sensation 
magnitude  of  the  test  stimulus  will  be  reduced. 
Furthermore,  in  studies  of  tactile  masking,  where  the 
accuracy  of  identification  of  complex  two-dimensional 
patterns  is  reduced  in  the  presence  of  a  masking  stimulus 
[12,  14],  error  analysis  shows  subtle  spatial  interactions 
between  pattern  and  mask  [11,  15],  Similarly,  Craig  [17] 
demonstrated  that  the  ability  to  localize  patterns  on  an 
array  was  influenced  by  the  presence  of  preceding  stimuli, 
especially  when  test  and  target  were  at  two  different 
locations.  Even  physiological  data  indicate  varying 
degrees  of  interaction  between  stimulated  loci  [23,  31]. 
These  findings  of  simple  and  complex  vibrotactile  spatial 
effects  warrant  further  exploration  of  the  bases  of  the 
interactions  that  could  affect  the  encoding  of  information 
from  tactile  communication  aids  and  devices. 


There  are  only  a  few  studies  of  tactile  resolution  in 
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which  the  ability  to  localize  vibrotactile  stimuli  has  been 
specifically  examined  (notably,  [58]).  In  attempting  to 
determine  the  appropriate  spacing  for  tactors  in  the  Tactile 
Vision  Substitution  System,  Eskildsen  [19],  explored 
simultaneous  and  successive  two-point  thresholds  on  the 
lateral  back  near  the  scapula  for  2  sec  60  pps  stimuli, 
finding  that  threshold  could  be  as  small  as  11  mm  for 
pairs  of  1  mm  contactors  (Weinstein’s  [63]  static  two- 
point  thresholds  were  c.  4  cm).  Rogers  [49]  examined 
spatial  resolution  on  the  fingertip,  with  two  constantly- 
vibrating  tactors.  For  frequencies  of  10  and  250  Hz, 
resolution  was  a  function  of  separation  with  poorest 
performance  at  2  mm.  Sherrick,  Cholewiak,  and  Collins 
[52]  used  carefully-controlled  25  or  250  Hz  stimuli  to 
appeal  to  high-  and  low-frequency  sensitive  cutaneous 
channels.  Observers  indicated  the  location  of  a  burst  of 
vibration  on  the  hypothenar  eminence  of  the  palm  (the 
edge  opposite  the  thumb).  Accuracy  was  generally  above 
75%,  with  low-frequency  stimuli  only  slightly  better 
localized.  The  absence  of  parametric  vibrotactile  spatial 
acuity  data  needs  to  be  addressed  directly,  in  order  for 
tactile  communication  systems  to  be  used  efficiently. 

1.2.  Pattern  perception  &  spatial  localization 

The  history  of  device  design  illustrates  that  the 
resolution  of  displays  has  rarely  been  determined 
empirically.  Rather,  designers  have  resorted  to  the  data 
published  in  Weinstein’s  oft-cited  1968  paper  that 
describes  error  of  localization  and  2-point  threshold 
obtained  over  many  body  site.  As  an  example,  Bliss  [4] 
carefully  analyzed  the  spatial  resolution  of  cameras  used 
to  record  text  for  direct-translation  aids  for  blind  persons, 
relating  the  resolution  of  the  device  to  the  spatial 
frequency  bandwidth  of  the  images  to  be  processed. 
However,  to  determine  the  resolution  of  the  vibrotactile 
output  device,  he  used  Weinstein’s  static  two-point  data  as 
a  measure  of  the  skin’s  acuity.  As  a  guide  to  the 
development  of  vibrotactile  patterns  and  devices,  these 
measures,  obtained  with  static  patterns,  are  being 
inappropriately  generalized  to  vibratory  stimuli.  The 
relationship  between  two-point  touch  thresholds  and 
performance  on  durative  patterns  such  as  Braille  cells 
seems  more  intuitive.  Indeed,  Stevens  [54]  argues  that  the 
reduction  in  static  tactile  acuity  with  age  they  measured 
has  serious  consequences  for  blind  persons  trying  to  read 
Braille  characters.  It  is  likely,  however,  that  the  active 
movement  of  the  fingers  over  the  cells  in  normal  Braille 
reading  adds  a  level  of  richness  to  the  tactile  pattern  that 
may  compensate  for  the  inherent  mismatch  between  static 
stimulus  and  aging  receptive  systems.  In  contrast, 
vibrotactile  arrays  in  devices  like  the  Tactaid,  Optacon, 
TVSS,  or  TSAS,  described  earlier,  cannot  be  actively 
explored.  These  have  tactor  arrays  designed  for  passive 
reception.  In  none  of  these  arrays  has  there  been  a 


quantitative  measure  of  their  ability  to  accurately  present 
vibrotactile  patterns  that  have  spatial  characteristics 
matching  those  of  the  application  site. 

The  accuracy  with  which  persons  can  process  tactile 
patterns,  given  appropriate  temporal  factors,  ultimately 
comes  down  to  an  interplay  between  the  spatial  acuity  of 
the  skin  and  the  spatial  resolution  of  the  patterns  and 
display.  Some  have  argued  that  the  poorer  performance 
often  seen  with  tactile  patterns  is  a  natural  consequence  of 
the  inferior  acuity  of  the  skin,  at  least  in  contrast  to  the 
visual  system  [41,  42].  Recent  tactile  data  have  shown 
differences  in  pattern  identification  performance  between 
old  and  younger  subjects  that  appear  to  be  directly  related 
to  poorer  spatial  resolution  in  the  older  group  [9, 10]. 

1.3.  Aging  &  spatial  localization 

We  will  also  be  examining  vibrotactile  pattern 
processing  in  elderly  persons  to  compare  their 
performance  against  that  from  the  younger  population. 
This  exploration  into  aging  is  warranted  for  several 
reasons.  Hearing  acuity  decreases  rapidly  after  the  age  of 
60  irrespective  of  specific  otological  disease  [20],  while 
88%  of  visually-disabled  persons  are  over  the  age  of  60 
[28].  Consequently,  understanding  the  capabilities  of  the 
sense  of  touch  in  the  older  individual  has  great 
importance,  since  it  may  often  be  the  only  alternative 
system  available  to  supplement  remaining  sensibility. 
Furthermore  the  study  of  tactile  pattern  processing  in  this 
group,  because  of  the  known  changes  in  the  peripheral 
nervous  system,  should  help  us  understand  the 
mechanisms  involved  in  tactile  perception,  and  show  how 
these  findings  might  be  incorporated  into  better  devices 
for  sensory  substitution  or  augmentation. 

Basic  research  on  processing  of  tactile  stimuli  has 
demonstrated  significant  changes  with  age  revealed  when 
a  single  touch  or  a  burst  of  vibration  is  presented  to  the 
skin  [53,  59,  60],  Typically  these  studies  have  been 
concerned  with  measuring  tactile  or  vibrotactile  thresholds 
[22,  46,  80].  In  most  cases  there  is  a  decline  in  the 
capacity  [30]:  thresholds  increase,  or  discriminative 
capabilities  are  poorer,  in  older  individuals.  A  few 
notable  studies  have  found  similar  declines  in 
performance  when  the  effects  of  aging  are  examined  in 
complex  measures  of  tactile  pattern  processing,  such  as 
haptic  or  form  exploration  [1,  13,  38],  spatial  acuity  [53]), 
or  temporal  processing  [57].  The  underlying  cause  of  the 
progressive  loss  has  usually  been  attributed  to 
physiological  changes  in  the  skin  itself  [53,  59]  or 
neurological  factors  [39,  43,  61].  The  functional  decline 
in  tactile  perceptual  skills  also  parallels  age-related 
anatomical  and  morphological  changes  in  the  skin  and  its 
receptors.  In  aging  skin,  the  size,  shape,  and  number  of 
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cells  in  the  upper  layers  become  quite  variable  and  glands 
often  atrophy  or  become  inactive  [44].  Dramatic  age- 
related  alterations  in  some  mechanical  properties  of  the 
skin  such  as  compliance,  surprisingly,  do  not  appear  to  be 
tied  to  reductions  in  tactile  sensitivity  [65].  Changes  with 
aging  in  the  receptors  themselves  have  also  been  fairly 
well  documented.  In  general,  while  the  number  decreases, 
structural  coinplexity  increases.  For  example,  there  are 
one-third  as  many  Meissner  corpuscles  in  some  body  sites 
in  70-year-old  individuals  as  in  20-year-old  persons  [50]. 
In  addition,  there  are  fewer  Pacinian  corpuscles,  and  those 
that  remain  undergo  dramatic  distortions  in  shape  and 
size,  while  Merkel’s  disks  and  free  nerve  endings  undergo 
less-obvious  changes  with  age  [7].  Because  these 
structures  and  others  have  been  related  to  different  aspects 
of  tactile  sensitivity  [5,  33],  a  correlation  would  be 
expected  between  these  changes  and  variations  in  tactile 
acuity.  Although  tactile  pattern  perception  requires  a 
combination  of  both  spatial  and  temporal  processing 
abilities,  we  are  primarily  studying  spatial  processing 
abilities  to  provide  a  measure  of  the  change  in  this  sensory 
capacity  in  older  subjects  relative  to  a  younger  population. 

But  how  well  does  the  skin  appreciate  spatial  stimuli? 
We  are  probably  familiar  with  illustrations  like  the  next 
one  -  showing  the  considerable  variation  in  sensitivity 
over  the  surface  of  the  body  for  simple  touches,  or 
pressure  stimuli  (from  Weinstein  [631). 


Figure  3.  Variation  in  pressure  threshold  over  the 
body  (From  Weinstein  [63]). 


Similar  pictures  have  emerged  when  pairs  of  such 
points  were  used,  but,  as  discussed  earlier,  there  is  are 


Nevertheless,  there  is  little  question  that  tactile  sensitivity 
varies  considerably  over  the  body’s  surface,  whether 
measured  by  one  or  two  touches.  But  what  about 
vibration?  As  we  saw  earlier,  the  patterns  of  activity 
resulting  from  vibration  on  and  in  the  skin  can  be  quite 
complex  and  widespread,  as  shown  in  Figure  1.  Such  a 
pattern  can  also  result  from  more  natural  stimuli  such  as 
rubbing  or  stroking  of  the  skin,  although  stretch 
components  are  also  introduced. 

So  what  happens  when  we  look  at  the  skin’s 
sensitivity  to  vibration?  Observers  judged  stimuli  that 

Figure  4.  Vibrotactile  thresholds  on  hairy  and 
glabrous  skin.  Circles  represent  3  sites  on  the  back. 
(After  Verrillo  [59]). 

varied  in  vibratory  frequency  over  the  normal  range  of 
sensitivity  of  the  skin  -  from  about  20  to  about  300  Hz,  of 
varying  intensity.  They  indicated  when  they  just  felt  the 
stimulus.  When  we  compare  the  results  on  the  palm  of  the 
hand  versus  the  arm  (smooth  versus  hairy  skin),  we  have 
seen  impressive  differences  (Figure  4).  Hairy  skin  is  less- 
sensitive  over  the  whole  range.  A  few  additional  data 
points  are  shown  in  the  graph  from  a  single  observer  in 
my  lab.  These  are  thresholds  taken  on  the  costovertebral 
angle  of  the  back.  Depending  on  where  one  places  the 
stimulator,  within  a  range  of  a  few  cm,  a  high-frequency 
dip  can  be  recorded  —  or  not!  These  data  suggest  the 
presence  of  different  underlying  physiological  systems  - 
receptor  channels  that  differ  across  body  sites,  within 
body  sites,  and  over  the  age  span. 

Physiological  data  certainly  support  these  notions  - 
40  years  ago,  Sato,  von  Bekesy,  and  others  indicated  that 
at  least  one  component  of  these  curves,  the  high-frequency 
dip  -  reflected  the  operation  of  one  known  receptor,  the 
Pacinian  Corpuscle  -  which  is  found  to  be  sparsely 
distributed  in  hairy  skin  (Figure  5).  Anatomical 

Figure  5.  Properties  of  2  deeper-lying  glabrous  skin 
receptor  structures  (after  Vallbo  [56]). 

exploration  has  shown  even  more  structures  that  could  be 
involved  in  tactile  sensitivity,  and,  more  recently,  these 
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have  been  related  to  specific  perceptual  experiences  when 
the  glabrous  skin  of  the  hand  was  stimulated  [48]. 

Figure  6.  Properties  of  2  superficial  glabrous  skin 
receptor  structures  (after  Vallbo  [56]). 

Other  research  has  shown  that  in  the  human  hand,  the 
superficial  receptor  structures  (Figure  6)  have  extremely 
high  densities  in  the  fingertip  skin,  much  greater  than  that 
of  the  deeper  structures.  Nevertheless,  recall  that  the  most 
dramatic  variations  in  sensitivity  were  in  the  high 
frequency  ranges  where  the  Pacinian  determines 
threshold.  Remarkably,  given  the  extraordinary  sensitivity 
of  this  receptor  system  (less  than  lp  at  250  Hz),  it  is  the 
deepest-lying  of  all  of  the  structures.  When  we  try  to 
decide  which  of  these  subserves  spatial  pattern  processing, 
single  unit  data  suggest  significant  differences  in  their 
ability  to  resolve  spatial  detail  when  presented  with 
patterns  even  as  simple  as  Braille  cells. 

2.  Spatial  localization  on  linear  arrays 

How  does  vibratory  spatial  sensitivity  vary  over  the 
surface  of  the  body?  There  are  no  data  in  the  literature 
like  those  we  saw  for  the  two-point  thresholds.  We  have 
collected  some  data  that  address  the  accuracy  of 
resolution  of  spatial  vibrotactile  stimuli  on  two  body  sites: 
the  forearm  and  the  trunk  of  the  body.  In  the  first  case, 
eight  subjects  were  asked  to  identify  7  locations  of  150- 
msec  bursts  of  100  Hz  or  250  Hz  vibration  on  a  linear 
array,  equated  in  perceived  intensity.  Stimulus  frequency 
was  varied  in  order  to  appeal  to  different  underlying 
receptor  systems,  since  they  also  have  been  reported  to 
have  different  spatial  sensitivities.  Examining  the 
distributions  of  correct  responses  and  errors  to  each  site  in 
the  first  session  of  5  blocks  of  70  identification  trials,  (in 
Figure  7),  we  see  that  when  sites  are  25  mm  apart, 
accuracy  is 


Figure  7.  Localization  of  sites  on  a  7-tactor  linear 
array  on  the  forearm;  25  mm  separation,  100  Hz 
(upper  graph)  and  250  Hz  (lower). 

quite  poor  for  this  task,  particularly  in  the  mid-range  of 
loci.  The  higher  accuracy  at  the  ends  of  the  range  has  a 
number  of  sources  we  could  discuss  (specifically  related 
to  “anchor  points”  [6]).  Remarkably  there  is  no 
meaningful  difference  between  the  functions  obtained  for 
the  two  frequencies. 

What  would  happen  if  we  expanded  the  separation 
and  used  fewer  tactors  (since  the  forearm  has  a  fixed 
length)?  The  next  sets  of  data  were  taken  from  one  of  our 
studies  in  which  the  100-Hz  stimuli  were  presented  at  4 
sites  separated  by  50  mm  to  younger  (college-age)  and 


Figure  8.  Localization  of  sites  on  a  4-tactor  linear 
array  on  the  forearm  by  students  (left)  and  seniors 
(right):  50  mm  separation,  100  Hz. 

older  (60+  yrs)  subjects.  The  results  plotted  as  a  confusion 
matrix  in  Figure  8,  indicate  that  students  were  more 
precise,  making  fewer  errors  particularly  for  sites  more 
than  50  mm  away  from  the  target.  Although  performance 
was  quite  good  overall,  the  students  were  particularly 
accurate  in  the  interior  sites.  These  studies  are  continuing, 
and  will  be  expanded  to  other  body  sites. 
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Finally,  we  examined 
the  accuracy  with  which 
spatial  vibrotactile  stimuli 
are  resolved  on  the  trunk 
of  the  body.  In  this  case, 
seven  subjects  were  asked 
to  identify  the  locations  of 
150-msec  bursts  of  250- 
Hz  vibration  presented  on 
a  belt  of  12  vibrators 
spaced  at  equal  intervals 
around  the  lower 
abdomen,  at  a  level  just 
above  the  umbilicus. 

Here,  trained  observers 
responded  on  a  3- 
dimensional  cylindrical  keyboard,  as  shown  in  Figure  9. 
They  were  instructed  to  encode  the  stimuli  (and  key 
positions)  as  though  they  represented  hours  on  the  clock 
face,  with  12  o’clock  located  at  the  navel  and  6  o’clock  at 
the  spine.  The  data  were  collapsed  over  two  sessions  of 
300  trials,  and  responses  plotted  as  before,  as  a  function 
of  the  stimulus  location,  in  Figure  10.  Again,  average 
performance  was  very  good,  but  note  that  optimal 
identification  occurred  at  what  might  be  defined  as 
“anchor  points”  at  the  front  (12  o’clock)  and  rear  (6 
o’clock)  of  the  trunk  -  the  navel  and  spine. 

Finally,  despite  the  apparent  accuracy  of  identification 

Figure  9.  Participant  working  with  the  12-site  3- 
dimensional  factor  keyboard. 

identification,  even  for  250-Hz  stimuli,  it  is  helpful  to 
examine  the  amount  of  information  transferred  in  each 
case.  In  fact,  analyses  of  the  data  represented  in  Figure  7 
indicate  that  for  250  Hz  stimuli,  overall  correct 
identification  was  only  at  the  56%  level,  with  only 


Figure  10.  Correct  responses  and  errors  by  response 
sites  as  a  function  of  locus  on  a  12-tactor  belt:  250  Hz. 


1.11  bits  of  information  transmitted  (representing  2.15 
tokens)  out  of  the  2.807  bits  potentially  available  in  the  7 
stimulus  tokens.  Similarly,  when  the  stimulus  frequency 
was  100  Hz,  overall  percent  correct  was  54%,  and  only 
1.01  bits  of  information  were  transmitted  (representing 
2.01  tokens).  When  12  factors  were  tested  (Figure  10), 
overall  percent  correct  was  75%,  and  2.52  bits  of 
information  were  transmitted  out  of  a  potential  3.58 
available  in  the  stimuli,  encoding  5.74  of  the  12  tokens. 

Further  studies  are  being  conducted  to  better  explore 
the  stimulus  parameters  that  might  influence  vibrotactile 
localization.,  including,  in  particular,  a  parametric 
examination  of  stimulus  frequency  and  tactor  separation. 
Our  intention  is  to  optimize  vibrotactile  localization  at 
each  site  so  as  to  determine  the  design  parameters  for 
arrays  used  in  cutaneous  communication  systems. 
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Abstract 

Since  the  importance  of  haptic  interaction  in  virtual  en¬ 
vironments  grows,  the  relevance  of  good  haptic  UI  widgets 
becomes  more  important  too.  This  papers  shortly  discusses 
a  framework  for  building  3D  widgets  in  general  and  more 
specific  3D  menus.  The  usage  of  these  menus  was  assessed 
in  a  formal  user  test.  The  surplus  value  of  using  haptic 
feedback  in  3D  widgets  and  the  difference  in  performance 
of  clicking  versus  pushing  were  evaluated.  This  experiment 
and  its  results  are  explained  in  this  paper. 


1.  Introduction  and  motivation  of  the  research 

Haptic  interaction  is  becoming  more  important  in  3D  en¬ 
vironments.  Since  the  user  typically  holds  a  haptic  device, 
such  as  the  PHANToM  [14],  in  the  dominant  hand,  it  is 
not  convenient  to  use  a  mouse  for  menu  interaction.  Al¬ 
though  a  minority  of  the  users  are  capable  of  manipulating  a 
mouse  with  the  non-dominant  hand,  it  is  likely  that  another 
3D  input  device,  such  as  a  space  mouse  is  used  for  cam¬ 
era  navigation.  Such  a  two-handed  input  metaphor  is  com¬ 
mon  in  many  3D  applications,  such  as  3D  modelling  [3, 15]. 
Furthermore,  a  menu  paradigm  should  be  developed  that 
also  supports  users  who  are  not  bimanual.  Therefore,  we 
chose  to  use  the  haptic  device  to  interact  with  a  menu.  This 
opens  the  possibility  for  placing  a  3D  menu  inside  the  vir¬ 
tual  world. 

Although  menu  interaction  has  been  thoroughly  explored 
in  (traditional)  VR  research  [2],  the  implications  and  most 
effective  use  of  3D  menus  in  a  haptic  context  are  not  yet 
known.  Different  approaches  to  incorporate  menus  in  a  vir¬ 
tual  environment  can  be  identified.  In  JDCAD  [6,  8]  Spher¬ 
ical  and  ring  menus  are  used,  while  pie  menus  are  utilized 
in  HoloSketch  [4].  The  latter  approach  can  also  be  found  in 


Alias|Wavefront’s  Maya  [7],  where  a  radial  menu,  called  the 
Hotbox  is  placed  around  the  user’s  mouse  cursor.  This  is, 
however,  a  2D  solution.  Lindeman  ct  al.  [9]  used  hand-held 
windows  to  provide  a  passive  feedback  for  manipulating  2D 
widgets  in  a  3D  world  with  the  user’s  finger. 

Active  force  feedback,  using  a  PHANToM  device,  has 
been  applied  in  various  2D  desktops.  Force  feedback  has 
been  incorporated  in  the  X  desktop  by  Miller  et  al.  [10,  11] 
while  Oakley  et  al.  enhanced  the  Microsoft  Windows  desk¬ 
top  [12].  Some  research  into  3D  haptic  user  interfaces  has 
has  also  been  conducted:  Anderson  et  al.  [1]  have  created 
a  interface  builder  for  graphical  and  haptic  user  interfaces 
(GHUIs)  for  FLIGHT  (FLIGHT  is  now  called  e-Touch  [5] ) 
in  which  several  interfacing  components  are  present.  Since, 
to  our  knowledge,  no  research  exists  in  which  the  usefulness 
of  haptic  menus  in  investigated,  we  developed  a  3D  haptic 
menu  and  performed  a  formal  user  experiment  in  order  to 
assess  the  effectiveness  of  our  solution. 

This  paper  presents  a  technical  overview  of  the  menus 
that  we  provide  in  our  haptic  environments  and  discusses 
the  user  experiment. 

2.  Haptic  interfacing  elements 

All  3D  interfacing  elements,  such  as  menus,  toolbars  and 
dialogs,  have  to  provide  a  common  functionality.  For  in¬ 
stance,  they  have  to  support  the  haptic  device  and  should 
allow  the  developer  to  place  descriptive  texts  on  them.  We 
call  these  objects  “haptic  UI  elements”. 

2,1.  Placement 

Another  problem  with  3D  UI  elements  is  the  fact  that 
they  sometimes  obscure  the  element  of  interest:  a  dialog, 
which  asks  to  confirm  the  deletion  of  an  object,  should  not 
be  placed  in  front  of  this  object.  Likewise,  if  an  object 
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Figure  2.  Opaque  menu 


were  to  obscure  a  haptic  U1  element,  the  haptic  UI  element 
would  be  unusable.  Therefore,  a  haptic  UI  element  should 
be  placed  rather  in  the  foreground  of  the  virtual  environment 
than  in  the  background.  However,  it  must  not  hinder  the 
user  in  interacting  with  the  virtual  objects.  Hence,  we  have 
developed  haptic  UI  elements  that  can  arbitrarily  be  posi¬ 
tioned  in  the  virtual  environment,  but  are  typically  placed 
in  the  foreground.  The  haptic  UI  elements  are  semitrans¬ 
parent,  thus  not  obscuring  the  virtual  objects  [16].  When  the 
virtual  pointer  approaches  the  haptic  UI  element,  it  fades  in 
to  become  opaque.  As  the  pointer  moves  away,  the  haptic 
UI  element  fades  out  again.  Figures  1  and  2  show  a  menu 
in  its  semitransparent  and  opaque  form. 

2.2.  Implementation 

In  order  to  use  haptic  UI  elements  in  a.  virtual  environ¬ 
ment,  we  have  written  an  abstract  C++  class,  which  supports 
the  above-mentioned  functionalities  for  3D  UI  interaction. 
When  the  first  haptic  UI  element  is  instantiated,  it  makes 
a  series  of  OpenGL  display  lists  in  which  the  alphabet  is 
stored.  All  haptic  UI  elements  can  then  use  these  display 
lists  to  display  their  textual  contents.  Likewise,  the  abstract 
class  is  responsible  for  placing  a  haptic  constraint  in  the 
virtual  environment.  This  haptic  constraint  represents  the 
haptic  UI  clement  and  calculates  where  the  virtual  pointer, 
representing  the  haptic  device’s  position  and  orientation,  is 
located  with  respect  to  the  haptic  UI  element. 

Currently,  a  class  is  derived  from  the  abstract  class, 
which  implements  a  3D  menu:  a  2D  menu  object,  which 
can  be  arbitrary  positioned  in  3D  space.  Each  menu  item 
in  this  3D  menu  is  indicated  by  a  text,  e.g.  “Exit”.  Option¬ 
ally,  a  descriptive  icon  can  be  depicted  in  front  of  the  menu 
item.  In  order  to  make  the  menu  recognizable  for  the  user, 
it  employs  the  Windows  colour  scheme.  Two  different  in¬ 
teraction  methods  arc  provided  by  the  menus:  “point  and 
click”  and  pushing  against  the  menu  item.  The  first  method 
makes  use  of  the  standard  method  for  accessing  3D  menu’s, 
although  a  3D  pointer  is  used  in  this  case,  while  the  latter  is 
based  on  real  life  interaction  with  buttons  and  switches. 


3.  User  experiment 

The  usefulness  of  the  menu  interactions  was  assessed  in 
a  formal  user  experiment.  We  wanted  to  assess  if  the  “point 
and  click”  metaphor  is  better  than  the  push  metaphor  and 
if  force  feedback  improves  menu  selection.  A  scene  repre¬ 
senting  a  room,  containing  a  cube  and  menu,  was  shown  to 
the  users.  The  menu  items  indicated  7  colours;  the  name  of 
each  colour  was  preceded  by  a  square  in  the  same  colour,  so 
that  the  user  could  easily  match  the  colour  of  the  cube  with  a 
menu  item.  We  choose  to  add  this  square  instead  of  depict¬ 
ing  the  text  in  the  colour  it  describes,  because  not  all  colours 
are  equally  readable.  Figure  3  depicts  the  experimental 
scene;  since  our  test  persons  were  native  Dutch  speakers, 
the  colours  are  indicated  in  Dutch.  The  test  subjects  had  to 
indicate  the  current  colour  of  the  cube,  which  was  randomly 
chosen,  in  the  menu.  The  performance  of  the  users  when  us¬ 
ing  haptic  feedback,  provided  by  a  PHANToM  device,  was 
compared  with  a  condition  where  no  haptic  feedback  was 
present.  In  each  condition,  a  test  person  was  presented  with 
5  practice  trials  and  15  measured  trials. 


Figure  3.  Experimental  scene 


Twenty-four  test  persons,  twenty  males  and  four  females 
with  an  average  age  of  30,  participated  in  a  counterbalanced 
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repeated  measures  design.  In  order  to  avoid  negative  trans¬ 
fer  effects,  the  test  persons  where  evenly  distributed  into 
two  groups  according  to  age,  sex  and  experience  with  com¬ 
puters.  One  group  of  12  test  subjects  was  presented  with 
the  “point  and  click”  interaction  (click  condition),  while  the 
other  group  had  to  push  against  the  menu  items  (push  con¬ 
dition). 

The  dependent  variables  we  measured  were:  elapsed 
time  (in  ms),  needed  to  select  a  menu  item,  the  distance 
covered  by  the  virtual  pointer  (in  mm)  and  the  number  of 
erroneous  selections.  The  elapsed  time  was  further  divided 
in  the  approach  time,  needed  to  come  in  the  vicinity  of  the 
menu  and  the  homing  time,  needed  to  select  the  item  within 
the  menu.  A  trial  was  considered  complete,  when  the  cor¬ 
rect  menu  item  was  selected.  After  completing  the  test  in 
each  particular  condition  the  subjects  were  asked  to  answer 
a  series  of  questions.  After  completing  the  second  condi¬ 
tion,  a  comparing  questionnaire  was  presented.  This  ques¬ 
tionnaire  presented  the  test  persons  some  questions  about 
their  subjective  feeling  of  performance  and  frustration. 

4.  Results 

The  results  for  both  conditions  were  analysed  using  two- 
way  ANOVA  over  all  measurements  (click  and  push  condi¬ 
tions).  This  analysis  reveals  that  the  use  of  force  feedback 
only  significantly  reduces  the  number  of  errors  that  were 
made  by  the  test  subjects.  When  no  force  was  applied,  the 
results  were  better  for  all  other  dependent  variables,  though 
not  significantly.  Table  1  summarizes  these  results. 


Table  1.  Statistical  analysis  of  all  results 


With  force 

P-value 

Avg. 

approach 

time 

2053.7 

2112.1 

.86 

Avg. 

homing 

time 

A 

3167.4 

4076.5 

.14 

Avg. 

total 

time 

5221.1 

6188.6 

.11 

Avg. 

distance 

60.7 

74.8 

.11 

Total 

errors 

150 

68 

<  .001 

Furthermore,  the  trade-off  between  precision  and  speed 
is  confirmed  by  our  results,  since  a  negative  correlation  be¬ 
tween  the  approach  and  homing  time  can  be  found  (see  ta¬ 
ble  2). 

However  if  the  two  selection  mechanisms  are  compared, 
a  significant  difference  can  be  found  in  favour  of  the  click¬ 
ing  with  the  stylus  switch  (see  table  3).  At  first  sight,  this 


Table  2.  Correlations 


Avg. 

approach 

time 

Avg. 

homing 

time 

Avg. 

distance 

Avg. 

approach 

time 

1.0 

Avg. 

homing 

time 

-0.27 

1.0 

Avg. 

distance 

0.27 

0.76 

1.0 

Total 

errors 

0.06 

0.40 

0.35 

seems  to  be  contra  intuitive:  in  real-life,  selecting  a  but¬ 
ton,  e.g.  a  button  on  a  microwave  oven,  is  done  by  push¬ 
ing  against  the  button,  not  by  clicking  with  another  device. 
However,  in  real-life  people  push  against  buttons  with  their 
fingers,  not  with  a  pen,  while  people  are  used  to  the  “point 
and  click”  interaction  when  working  with  a  computer.  As 
an  alternative,  we  could  have  chosen  to  use  the  PHANToM 
thimbal,  because  it  allows  the  user  to  push  against  objects 
with  the  index  finger.  However,  most  of  our  applications 
require  6DOF  input,  which  can  only  be  provided  with  the 
encoder  stylus.  Furthermore,  most  3D  interaction  benefits 
from  having  a  switch  on  the  encoder  stylus  (e.g.  to  select 
objects  in  the  virtual  world). 


Table  3.  Comparing  the  click  with  the  push 
condition 


Click 

Push 

P-value 

Avg. 

approach 

time 

1387.6 

2778.2 

<  .001 

Avg. 

homing 

time 

1894.9 

5348.9 

<  .001 

Avg. 

total 

time 

3282.5 

8127.1 

<  .001 

Avg. 

distance 

42.7 

92.8 

<  .001 

Total 

errors 

79 

137 

<  .001 

Within  the  click  condition,  no  significant  difference  was 
found  in  the  time  needed  to  complete  the  condition  with 
haptic  feedback  and  the  condition  without  haptic  feedback. 
The  approach  time  was  slightly  better  when  no  haptic  feed¬ 
back  was  present.  However,  this  was  compensated  by  the 
better  homing  time  in  the  condition  with  haptic  feedback, 
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leading  to  better  over  all  performance  in  the  condition  with 
haptic  feedback.  Video  recordings,  from  the  user  test,  sug¬ 
gest  that  this  difference  is  caused  by  a  slight  hesitation 
when  approaching  the  haptic  menu  because  the  test  subjects 
searched  the  haptic  plane,  whereas  in  the  condition  without 
haptic  feedback,  the  test  subjects  just  located  the  pointer  in 
the  vicinity  of  the  menu.  However,  as  soon  as  the  virtual 
pointer  hit  the  haptic  plane,  the  test  subjects  could  use  this 
plane  as  a  guide  and  worked  faster  and  more  precise.  The 
fact  that  haptic  feedback  aids  the  user  in  being  more  effi¬ 
cient  is  supported  by  the  results  of  the  error  measurements 
(see  table  4).  In  both  cases  180  (12  test  persons  x  15  trials) 
correct  clicks  were  recorded.  In  the  condition  without  hap¬ 
tic  feedback  another  63  erroneous  errors  were  made;  hence, 
26%  of  all  clicks  were  wrong.  Haptic  feedback,  however 
reduces  this  figure:  8.2%  of  all  clicks  (16  out  of  196)  were 
erroneous.  This  result  is  statistically  significant.  We  reckon 
that  this  difference  is  caused  by  test  persons,  who  overshoot 
the  virtual  pointer  when  no  haptic  feedback  is  present.  Fur¬ 
thermore,  if  the  encoder  stylus  does  not  rest  against  a  virtual 
plane,  clicking  on  the  switch  will  sometimes  cause  the  sty¬ 
lus  to  move  downwards  before  the  click  is  registered. 


Table  4.  Analysis  of  the  click  condition 


Without  force 

With  force 

P-value 

Avg. 

approach 

time 

1365.3 

1409.8 

0.82 

Avg. 

homing 

time 

2038.4 

1751.4 

0.35 

Avg. 

total 

time 

3403.8 

3161.3 

0.41 

Avg. 

distance 

42.8 

42,5 

0.94 

Total 

errors 

63 

16 

<  .001 

No  significant  difference  was  found  in  the  results  of  the 
post  experiment  questionnaire,  although  a  number  of  test 
subjects  in  the  push  condition  were  visibly  frustrated  during 
the  experiment. 

5.  Discussion 

Our  experiment  shows  that  selection  of  a  menu  item  in 
a  haptic  scene  can  be  best  performed  by  using  a  “point  and 
click”  metaphor.  We  believe  that  these  results  are  valid  for 
a  number  of  selection  widgets,  such  as  toolbars  and  lists.  In 
our  setup,  pointing  can  be  accomplished  in  a  reliable  and 
fast  manner.  It  is  reliable  because  the  menu  item  that  can  be 
selected  lights  up  when  the  pointer  is  near  to  the  item  (but 


this  is  also  true  for  the  pushing  mechanism).  It  is  also  fast, 
because  the  user  does  not  have  to  touch  the  menu,  light¬ 
ing  a  menu  item  by  coming  in  the  vicinity  is  good  enough. 
However,  the  menu  lets  the  users  rest  their  hand  against  a 
haptic  plane,  which  helps  to  reduce  errors.  We  noticed  both 
behaviours  during  the  test. 

Of  course,  the  haptic  case  still  does  not  provide  perfect 
selection,  so  usual  error  correction  mechanisms,  such  as  a 
confirmation  for  critical  operations  (e.g.  clearing  the  virtual 
scene)  and  an  undo  mode  for  other  operations,  must  be  pro¬ 
vided. 

A  problem  that  arose,  when  working  with  the  haptic 
menu  was  the  fact  that  it  was  only  touchable  from  the  front. 
We  allowed  to  push  trough  the  menu,  when  approaching 
it  form  the  back  in  order  to  enhance  the  push  condition. 
Sometimes,  the  test  subject  would  get  lost  and  ended  up 
behind  the  menu.  In  order  to  push  against  the  menu,  the 
user  must  be  in  front  of  it.  In  this  case  the  test  person  would 
have  to  go  round  the  menu  with  the  pointer  If  the  back  of  the 
plane  would  also  be  touchable.  We  choose  to  keep  this  ef¬ 
fect  in  the  “point  and  click”  condition,  in  order  to  minimize 
the  differences  between  both  conditions.  Tire  test  persons 
in  the  push  condition  where  happy  with  this  feature,  but  the 
test  persons  in  the  “point  and  click”  condition  that  ended  up 
behind  the  menu  reported  that  they  would  be  more  efficient 
if  the  menu  would  be  touchable  from  two  sides.  This  is  an 
extra  argument  in  favour  of  the  “point  and  click”  condition, 
since  the  use  of  a  menu  that  is  touchable  from  both  sides 
is  consistent  with  a  haptic  virtual  environment.  However, 
further  research  is  needed  to  support  this  thesis. 

Although  the  usage  of  semi -transparency  allows  us  to 
place  the  menu  in  an  arbitrary  location,  it  would  be  better  if 
it  would  not  be  visible  if  the  user  does  not  want  to  use  the 
menu.  We  have  already  explored  the  use  of  head-tracking  in 
a  desktop  virtual  environment  [13J.  With  this  setup,  users 
can  virtually  enlarge  their  workspace  and  can  look  at  the 
virtual  world  from  different  viewpoints  in  an  intuitive  man¬ 
ner.  If  a  menu  would  be  placed  just  outside  of  the  users 
view,  than  the  menu  can  be  easily  accessed  without  being 
intrusive.  Of  course,  semi -transparency  would  still  be  de¬ 
sirable,  because  it  is  still  possible  that  the  user  just  wants  to 
examine  an  object. 

6.  Conclusions 

As  a  first  conclusion,  we  can  state  that  haptic  3D  menus 
in  a  virtual  environment  should  use  the  “point  and  click” 
metaphor  for  selection  and  not  a  pushing  metaphor  when 
using  an  encoder  stylus,  although  the  latter  seems  to  be  ob¬ 
vious  from  real-world  experiences.  Secondly,  haptic  feed¬ 
back  is  invaluable  in  this  interaction,  in  order  to  reduce  the 
number  of  errors  that  arise  due  to  overshooting.  We  believe 
that  similar  conclusions  can  be  drawn  for  other  haptic  UI 
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elements,  such  as  dialogs  and  toolbars,  although  further  re¬ 
search  is  needed  to  support  this  hypothesis. 
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Abstract 

Users  of  collaborative  systems  are  typically  restricted  to 
communication  through  voice  and  video  links.  Users  find  this 
difficult  -  it  does  not  encompass  the  richness  of 
communication  they  are  accustomed  to  in  the  real  world. 
Attempting  to  address  this  problem  we  describe  the 
implementation  of  a  novel  mechanism  for  haptic 
communication  based  around  interactions  between  users’ 
cursors.  An  initial,  and  mainly  observational,  evaluation  is 
described,  along  with  some  promising  results.  We  show 
improvements  in  subjective  experience  and  suggest  several, 
more  formal,  avenues  for  future  research. 

1.  Introduction 

With  the  advent  of  the  World  Wide  Web  (WWW), 
distributed,  collaborative  technologies  have  begun  to  gather 
momentum.  Real-time  collaborative  tools  are  becoming  more 
common,  and  there  is  growing  acceptance  that  such  tools  may 
be  able  to  reduce  travel  costs  and  facilitate  group  and  team 
based  work.  However,  these  systems  still  face  problems. 
Crucially,  the  communication  that  takes  place  between  users 
does  not  reflect  the  richness  of  communication  in  the  real 
world.  Even  assuming  high  quality  video  and  audio  links, 
information  pertaining  to  sources  such  as  bodily  gestures,  eye 
gaze,  and  background  noise  is  typically  lost. 

Consequently,  users  of  these  systems  experience  difficulty 
communicating  effectively  with  one  another.  Users  typically 
find  it  difficult  to  maintain  awareness  of  the  actions  of  others, 
and  to  meaningfully  coordinate  their  work  activities.  Users 
find  it  hard  to  infer  where  a  co-worker’s  attention  is  directed, 
and  the  use  of  gestures,  both  as  aids  to  conversational  token 
passing  and  as  indicators  of  topic  or  information,  is  less 
effective  than  interactions  in  the  real  world. 

There  is  a  large  body  of  research  that  attempts  to  address 
these  issues.  Much  of  it  focuses  on  the  addition  of  information 
through  novel  means.  There  is  extensive  work  investigating 
the  impact  of  video  systems  that  allow  participants  to 
maintain  eye  contact  [12],  and  various  researchers  have 
looked  at  the  possibilities  of  displaying  images  of  the  hands 


and  arms  of  remote  users  [8].  Audio  information,  in  the  form 
of  auditory  icons  [4],  has  also  been  shown  to  be  an  effective 
addition  to  these  systems.  However,  despite  the  improvements 
suggested  by  this  research,  the  problems  faced  by 
collaborative  systems  still  remain. 

Here  we  consider  how  haptics  could  be  used  to  address 
these  problems.  Previous  literature  concerning  haptic 
communication  in  dedicated,  standalone  devices  has 
suggested  that  it  provides  a  strong  interpersonal  link  between 
users,  and  raises  levels  of  presence  [2,  3].  More  recent  work 
investigating  the  cooperative  performance  of  physical  tasks  in 
virtual  environments  has  shown  significant  performance 
benefits  and  an  increase  in  the  sense  of  “togetherness”  and 
presence  achieved  between  participants  [1,  15]. 

However  the  majority  of  collaborative  tasks  are  not 
physical.  Here  we  describe  an  approach  that  attempts  to  take 
the  benefits  of  haptic  communication  -  not  only  increases  in 
performance,  but  perhaps  more  importantly,  a  greater  sense  of 
engagement  and  presence  -  and  apply  and  evaluate  their 
effects  in  a  realistic  collaborative  task.  We  focus  on  trying  to 
augment  the  communication  between  users  of  synchronous 
shared  editors.  Shared  editors  are  a  common  collaborative 
tool  and  allow  multiple  users,  often  situated  at  different 
locations,  to  simultaneously  work  on  the  same  document. 

2.  Implementation 

2.1.  Haptic  Communication 

Synchronous  shared  editors  typically  include  graphical 
avatars,  known  as  telepointers.  These  are  additional  cursors 
used  to  represent  the  position  of  other  users  in  the  workspace. 
In  accordance  with  the  haptic  communication  described  by 
Oakley  et  al.  [13],  we  base  our  haptic  communication  on 
enhancing  interactions  between  these  cursors.  Oakley  et  al. 
define  five  different  mechanisms  for  haptic  cursor 
communication.  Firstly,  a  push  effect,  which  transforms 
cursors  from  a  purely  graphical  representation  into  tangible 
objects  -  they  can  be  felt,  knocked  into  and  pushed  around. 
Secondly,  a  gesture  effect  which  allows  cursors  to  pull  each 
other  around.  One  user  can  pick  up  another  user,  causing  that 
user  to  be  constrained  to  follow  an  accurate  representation  of 
his  or  her  path.  Thirdly  and  fourthly,  locate  and  grab  effects, 
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which  let  users  activate  forces  to  guide  themselves  towards 
another  user  and  to  guide  other  users  towards  themselves. 
These  effects  differ  from  the  gesture  effect  in  that  they  make 
no  attempt  to  represent  a  path,  merely  a  position.  Finally 
Oakley  et  al.  describe  a  proximity  effect,  which  varies  the 
viscosity  of  workspace  according  to  the  proximity  of  other 
users,  and  is  designed  to  provide  a  basic  sense  of  the  presence 
of  others. 

2.2.  Haptic  Simulation 

The  haptic  simulation  was  implemented  in  C++  under 
Windows  NT  and  for  the  PHANToM  (from  SensAble 
Technologies).  As  we  focused  on  desktop  editing  tasks  the 
PHANToM  was  given  control  of  the  cursor  to  provide  a 
standard  pointer  interface.  However,  as  the  PHANToM  is  an 
absolute  position  device,  there  were  some  discrepancies  with 
normal  mouse  operation.  The  most  important  of  these  was 
that  stiff  walls  lined  the  edges  of  the  PHANToM’s  range, 
preventing  users  from  further  motion.  The  PIIANToM’s 
button  was  mapped  to  a  left  mouse  click,  The  workspace  itself 
was  presented  as  a  vertical  plane. 

The  gesture,  locate  and  grab  effects  all  require  an  explicit 
command  to  initiate  and  halt.  All  commands  were  mapped  to 
the  PHANToM’s  single  button.  To  gesture,  a  user  moved 
over  another  user  and  depressed  the  button;  to  end  the  gesture 
the  button  was  released.  This  control  mechanism  was 
designed  to  support  the  metaphor  of  taking  hold  of,  and 
pulling  along,  another  user.  The  locate  effect  was  activated  by 
pulling  backwards  from  the  workspace  and  depressing  the 
button.  Releasing  the  button  stopped  the  effect.  Finally  the 
grab  effect  was  initiated  by  pulling  backwards  from  the 
workspace,  performing  a  double  click  and  keeping  the  button 
down  after  the  second  click.  Once  more,  releasing  the  button 
slopped  the  effect.  All  other  buttons  presses  were  passed  to 
the  system  as  normal  mouse  events.  The  rational  for  using  the 
PFIANToM’s  button  in  this  way  was  to  ensure  that  the  haptics 
could  only  be  activated  while  the  PHANToM  was  being  held. 

2.3.  Implementation  of  Networked  Haptics 

Haptic  feedback  in  general  requires  high  update  rates, 
typically  of  500  Hz  or  more  [11].  Most  modern  networks,  for 
instance  ethernet  or  the  Internet,  do  not  provide  a  quality  of 
service  at  anywhere  near  this  level.  Several  researchers  are 
working  to  address  this  issue  [16].  The  goal  of  this  research, 
however,  is  to  investigate  the  potential,  for  a  user,  of 
collaborative  haptics.  Consequently,  a  high  performance, 
loss-less  transmission  medium  was  required.  To  achieve  this 
we  first  analyzed  the  refresh  rate  requirements  for  the  haptic 
cursor  communication. 

In  general,  high  update  rates  are  required  to  support  the 
production  of  stable  stiff  objects  [1 1]  and,  in  the  haptic  cursor 
communication,  only  the  push  effect  involves  the  production 
of  an  object  at  all  -  the  other  cursor.  The  proximity  effect 
does  not  require  any  such  representation,  merely  modifying 
the  viscosity  of  the  workspace  based  on  the  distance  to  other 


cursors,  and  as  such  should  be  fairly  insensitive  to  low  update 
rates.  Changes  in  the  viscosity  from  one  millisecond  to 
another  are  relatively  unimportant  and  should  fall  well  below 
the  perceptual  threshold.  The  locate,  grab  and  pull 
interactions  all  regard  the  other  user’s  position  as  the  target  of 
a  homing  force  and  as  such,  when  the  range  to  the  target  is 
small,  so  is  the  magnitude  of  the  targeting  force.  Lower  forces 
can  be  adequately  represented  by  lower  update  rates  [11]. 
Conversely  when  the  distance  to  the  target  is  large,  the 
targeting  force  is  also  relatively  large.  However  in  this 
instance  the  distance  that  the  target  must  move  to  cause  a 
substantial  change  in  the  targeting  force  is  also  large.  These 
factors  all  combine  to  make  the  haptic  communication 
relatively  tolerant  of  low  update  rates. 

Furthermore,  an  analysis  of  the  total  network  demands  of 
haptic  cursor  communication  leads  to  the  conclusion  that  even 
assuming  a  high  update  rate,  a  low  total  bandwidth  is 
sufficient.  This  is  because  the  only  remote  information  of 
significance  to  the  simulation  is  the  position  and  state  of  other 
cursors  -  for  instance  whether  they  are  engaged  in  a  gesture. 
This  can  be  expressed  in  a  handful  of  bytes  (16  bytes  per 
PHANToM  per  update  in  the  current  implementation). 

Finally,  to  further  simplify  development  some  of  the 
requirements  of  a  collaborative  system  were  relaxed.  Firstly 
we  choose  to  consider  a  system  consisting  of  only  two  users 
and  secondly  we  decided  to  implement  the  communication 
through  a  dedicated  streaming  connection  rather  than  across  a 
network.  Both  of  these  choices  are  commonly  seen  in 
groupware  research  [e.g.  4,  8]  and  typically  reflect  the 
intention  of  the  work  -  to  evaluate  the  potential  of  the 
communication  for  a  user,  not  to  investigate  underlying 
network  performance. 

With  these  analyses  in  mind,  the  haptic  communication 
was  implemented  running  over  serial  cables  and  providing  a 
dialogue  between  two  machines  at  100  Hz.  This 
implementation  suffers  from  the  restrictions  of  being  simple 
for  two  users,  but  more  complex  for  more,  and  of  being 
confined  to  machines  positioned  only  a  few  metres  from  one 
another.  Due  to  the  update-tolerant  nature  of  the  cursor 
communication,  this  refresh  rate  provides  a  subjective 
experience  that  is  both  stable  and  of  an  acceptable  quality. 

2.4.  Collaborative  Editor 

To  provide  a  flexible  platform  for  evaluating  this  haptic 
communication,  we  turned  to  GroupKit  [14],  a  high  level 
tcl/tk  based  groupware  toolkit.  GroupKit  provides  support  for 
the  management,  including  the  creation,  manipulation  and 
mediation  of  access  to,  shared  information,  and  basic 
groupware  aids  such  as  telepointers. 

The  product  of  coupling  the  haptic  communication  with 
GroupKit  is  CHASE  (Collaborative  Haptics  And  Structured 
Editing),  a  synchronous,  Relaxed  What  You  See  Is  What  I 
See  (RWYSIWIS)  structured  drawing  tool,  pictured  in  figure 
1 .  It  provides  telepointers  and  allows  users  to  simultaneously 
work  on  a  large  canvas  while  each  maintaining  a  separate 
view  of  it.  CHASE  allows  users  to  create  and  edit  four  types 


Eurohaptics  2001  Birmingham  July  2001 


of  object:  text  items,  rectangular  groups,  oval  groups,  and 
links.  Text  items  can  be  placed  in  group  objects  and  links  can 
be  made  between  them.  All  items  can  be  freely  moved,  edited, 
resized  and  otherwise  manipulated. 

One  consequence  of  the  combination  of  the  haptic  cursor 
communication  and  a  collaborative  RWYSIWIS  workspace  is 
that  users  can  feel  forces  that  attempt  to  move  them  outside  of 
their  current  view  of  the  workspace.  To  resolve  these  forces 
intelligibly,  the  haptic  workspace  was  restricted  to  only  allow 
cursor  movement  within  the  window  of  the  GroupKit 
application;  walls  were  presented  at  the  edge  of  the  CHASE 
window.  When  users  pushed  into  these  walls,  as  they  would  if 
forces  were  pulling  them  off  the  workspace,  the  workspace 
scrolled  in  the  direction  that  they  are  pushing,  gradually 
changing  their  view  until  their  target  was  on  screen.  This 
solution  had  the  consequence  of  providing  a  new  mechanism 
for  scrolling  -  simply  pushing  into  the  walls  of  the 
workspace. 

The  haptic  simulation  communicated  with  CHASE  through 
mouse  events  and  also  through  a  socket  connection.  The 
mouse  events  formed  the  main  part  of  the  interface,  and 
functioned  simply  as  a  ^consequence  of  using  the  PHANToM 
as  a  cursor  control  device.  The  socket  connection  was  used  to 
communicate  all  other  information,  such  as  the  scroll  position 
of  the  CHASE  workspace,  and  to  transmit  scrolling  events 
when  the  PHANToM  pushed  into  a  wall.  A  diagram  of  this 
architecture  is  shown  in  figure  2. 

3.  Evaluation 

3.1.  Design 

Evaluation  in  collaborative  systems  is  a  more  challenging 
process  than  the  evaluation  of  single  user  systems  [6]. 
Frequently  a  metric  such  as  time  to  complete  a  task,  or  even 
quality  of  output,  is  less  important  than  the  somewhat 
ephemeral  “quality”  of  the  communication  between  the  users. 
Furthermore,  the  use  of  collaborative  systems  is  arguably 
much  more  dependant  on  context  than  single  user  systems. 
There  is  less  guarantee  that  a  feature  which  appears  to  work 
successfully  in  a  laboratory  situation  will  transfer  that  success 
to  real  world  use.  * 

This  evaluation  was  also  further  restricted  by  the 
preliminary  state  of  this  work.  Lacking  foundations  on  which 
to  build,  and  combined  with  the  challenge  presented  in  the 
evaluation  of  any  collaborative  software,  we  decided  that  the 
most  important  issue  to  address  was  one  of  acceptability.  We 
choose  to  address  the  question  of  whether  or  not  users  would 
adopt  this  novel  form  of  communication  and,  if  they  did  adopt 
it,  what  effects  it  would  exert  on  their  subjective  experience. 
Furthermore,  although  the  haptic  communication  was 
designed  with  a  purpose  in  mind,  we  felt  that,  if  users  did 
adopt  it,  they  were  likely  to  develop  unforeseen  uses  for  it.  A 
secondary  goal  was  simply  to  detect  any  novel  uses  of  the 
communication.  Finally,  as  this  software  represents  a 
prototype,  we  wanted  to  discover  any  problems  users 
experienced  with  the  communication. 


Figure  1 .  Telepointers  and  structured  objects  in  CHASE 

Designing  our  initial  evaluation  to  reflect  these 
considerations  we  chose  to  perfoim  an  observational  study  of 
users  performing  a  high  level  task.  This  allowed  us  to  observe 
whether  or  not  participants  adopted  the  communication,  what 
uses  they  put  it  to,  and  the  problems  they  had  with  it. 

3.2.  Participants 

Sixteen  users,  all  computing  science  students,  in  eight 
pairs,  participated  in  the  experiment.  Four  pairs,  forming  the 
Haptic  condition,  solved  a  problem  with  the  aid  of  the 
collaborative  haptic  feedback;  four  pairs,  the  Visual 
condition,  worked  without  it.  Both  sets  used  the  PHANToM 
in  the  same  restricted  workspace,  and  had  access  to  the  same 
novel  scrolling  technique.  None  had  previously  used  a  shared 
editor,  nor  had  anything  more  than  trivial  experience  with 
haptic  interfaces. 

3.3.  Task 


CHASE  was  used  as  a  simple  CASE  (Computer  Aided 
Software  Engineering)  tool.  Participants  were  required  to  read 
a  problem  statement  and  design  a  set  of  Unified  Modeling 
Language  (UML)  diagrams  to  solve  it.  The  advantages  of 
using  CASE  are  that  it  is  an  established  domain  for 
collaborative  tools  (e.g.  [5]),  and  that  problems,  ideal 
solutions,  and  semi-expert  users  are  easy  to  find  in  an 
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academic  environment.  While  CHASE  cannot  represent  all 
the  objects  in  the  UML  specification,  it  does  allow  the 
development  of  a  basic  diagram. 

3.4.  Measures 

Users  were  observed,  with  both  audio  and  video  recorded, 
throughout  the  experiment.  No  formal  dialogue  analysis  of 
this  data  is  currently  planned.  Four  questionnaires  were 
administered  at  the  end  of  the  experiment.  Firstly,  NASA 
TLX  [7],  an  established  questionnaire  designed  to  measure 
subjective  workload.  Secondly,  QUIS  [10],  a  standard 
questionnaire  for  assessing  the  usability  of  computer  systems. 
Thirdly,  the  ITC  Presence  Questionnaire  [9],  designed  to 
measure  presence  in  virtual  environments.  The  final 
questionnaire  was  created  specifically  for  this  experiment  to 
assess  collaboration.  It  consisted  of  ten  items,  to  be  rated  on 
seven-point  scales  (the  questions  are  included  as  part  of 
Figure  6,  in  the  results  section).  The  use  of  this  custom 
questionnaire  reflects  the  lack  of  an  accepted  tool  for  this 
purpose.  This  questionnaire  data  was  intended  to  allow  us  to 
gain  a  measure  of  the  influence  of  the  haptic  communication 
on  subjective  experience.  Objective  measures  of  time  to 
complete  task  and  quality  of  model  were  also  gathered,  but 
little  weight  has  been  attached  to  them. 

3.5.  Procedure 

For  simplicity  participants  were  seated  in  the  same  room, 
separated  by  a  screen.  They  could  not  see  one  another,  and  no 
video  link  was  provided.  They  could  talk  freely.  A 
disadvantage  of  this  setup  is  that  audio  information  from  the 
environment,  such  as  the  sound  of  keys  presses,  which  is  not 
typically  passed  between  subjects  at  different  locations,  was 
present.  Participants  in  both  conditions  went  through  an 
extensive  training  phase.  A  manual  to  CHASE,  and  in  the 
case  of  the  Haptic  condition,  to  the  haptic  cursor 
communication  was  provided  and  each  feature  explained  and 
demonstrated.  Participants  were  then  required  to  copy  a 
printed  UML  diagram  into  CHASE.  At  this  stage  it  was 
clearly  explained  that  the  example  diagrams  consisted  of 
acceptable  UML  constructs.  The  recording  equipment  was 
then  switched  on  and  subjects  then  began  to  solve  the  UML 
problem.  There  was  no  time  limit  imposed  on  the  task; 
subjects  were  instructed  to  stop  when  they  believed  the 
problem  had  been  solved  satisfactorily.  This  typically  took  an 
hour. 

4.  Results 

4.1.  General  Observational  Results 

There  was  substantial  variation  in  the  use  of  the  haptic 
communication.  One  pair  of  users  in  the  Flaptic  condition  did 
not  use  the  communication  at  all,  while  others  embraced  it, 
using  the  various  effects  regularly.  This  may  reflect  the  fact 
that  touch  is  a  very  personal  sense.  Individual  differences  and 


social  factors  may  well  exert  a  strong  influence  over  the 
adoption  and  use  of  this  kind  of  communication.  Furthermore 
it  seems  likely  that  communication  of  this  sort  breaks  new 
social  ground.  Most  users  prefixed  use  of  the  haptic 
communication  with  a  verbal  warning,  even  when  the 
communication  would  not  affect  the  other  user,  as  with  the 
locate  effect.  Users  have  not  previously  been  able  to 
communicate  in  this  way  and  appear  unsure  what  protocols 
should  mediate  their  interaction. 

Users  also  appeared  to  find  the  interface  to  the  haptic 
communication  difficult.  We  attribute  this  to  the  fact  that  the 
communication  was  completely  controlled  through  the  haptic 
device;  it  was  simply  overloaded.  A  final  crucial  point  is  that 
the  majority  of  the  users  appeared  to  find  the  haptic 
communication  engaging  and  helpful,  rather  than  annoying  or 
intrusive. 

4.2.  Observations  of  Locate  and  Grab  Effects 

The  majority  of  the  participants  in  the  haptic  condition 
immediately  understood  the  purpose  and  applicability  of  the 
locate  and  grab  effects.  They  used  the  locate  effect  regularly 
and  the  grab  effect  more  infrequently.  Reflecting  the  fact  that 
users  found  it  easier  to  find  one  another,  pairs  in  the  Haptic 
condition  tended  to  use  much  more  of  the  available  screen 
space  and  created  diagrams  that  spread  over  a  larger  area  than 
pairs  in  the  Visual  condition.  Diagrams  in  the  Visual 
condition  tended  to  be  very  compact,  with  different  sections 
abutting  one  another. 

Participants  in  the  Visual  condition  also  used  many 
different  techniques  for  finding  one  another  or  specific 
objects,  which  were  mainly  absent  from  the  Haptic  condition. 
They  would  describe  their  position  by  references  to  the 
position  of  the  scrollbars  at  their  location,  or  by  naming 
nearby  objects  or  diagrams.  One  participant  in  the  visual 
condition  occasionally  instigated  a  more  extreme  solution. 
Upon  finding  the  other  user’s  telepointer  he  would  endeavour 
to  maintain  a  view  on  it  for  as  long  as  possible  through  rapid 
scrolling  -  giving  the  appearance  of  pursuing  the  other  user. 

One  pair  in  the  Visual  condition  did  produce  a  diagram 
that  occupied  a  large  and  diverse  area  of  the  screen.  They 
suffered  from  considerable  confusion  while  discussing  where 
to  begin  new  elements  of  their  solution,  and  in  keeping  track 
of  one  another  when  they  came  to  review  their  diagrams.  Such 
confusion  was  far  less  evident  in  the  Haptic  condition. 

4.3.  Observations  of  Gesture  Effect 

The  haptic  gesture  was  used  infrequently.  Graphical 
telepointer  gestures,  such  as  when  a  shape  is  described  by 
simply  moving  along  its  contours,  or  more  commonly,  when 
an  object  is  indicated  by  moving  over  it,  were  far  more 
prevalent.  This  is  probably  due  to  the  fact  that  there  is  a  time 
cost  associated  with  the  haptic  gesture.  A  haptic  gesture 
involves  moving  over  another  user,  picking  that  user  up,  then 
moving  back  to  the  item  of  interest.  The  haptic  gesture  also 
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provides  little  enhancement  of  the  most  common  use  of  a 
graphical  gesture:  indicating  a  single  object. 

However,  the  haptic  gesture  was  used  in  more  complex 
situations.  One  pair  of  participants  used  the  haptic  gesture  to 
indicate  several  objects,  spread  over  an  area  of  the  workspace 
too  large  to  be  effectively  displayed  with  a  graphical  gesture. 
This  could  indicate  that  the  haptic  gesture  is  useful  for 
illustrating  complex  sets  of  data.  Another  pair  of  participants 
used  the  gesture  in  an  entirely  novel  way  -  one  user  selected  a 
group  object  that  they  were,  discussing  and  the  other  user  then 
began  to  gesture  to  the  first  user,  steering  them  (and 
consequently  the  object)  towards  what  they  thought  was  an 
appropriate  location. 

4.4.  Observations  of  Proximity  and  Push  Effects 

Observing  any  direct  usage  of  the  proximity  and  push 
effects  was  challenging,  as  neither  requires  any  explicit  action 
to  initiate,  nor  causes  an  observable  motion.  However  the  two 
effects  may  have  combined  to  increase  a  feeling  of  presence 
in  the  workspace.  Users  appeared  to  be  more  confident  about 
using  graphical  gestures  in  the  Haptic  condition,  which  we 
suggest  may  have  stemmed  from  an  increased  sense  of 
presence  brought  about  by  the  more  tangible  representation 
provided  by  haptic  communication. 

4.5.  Questionnaire  Results 

All  analyses  were  conducted  using  two  sample  between 
groups  t-tests.  Figure  3  contains  the  results  from  the  TLX 
questionnaire.  Overall  workload  did  not  significantly  change 
between  the  conditions.  The  Haptic  condition,  however,  was 
significantly  more  Physically  Demanding  than  the  Visual 
(p<0.05).  The  difference  in  the  Frustration  Experienced  factor 
approached  significance  (p<0.065)  and  we  attribute  this  trend, 
and  possibly  also  the  increase  in  Physical  Demand,  to  the 
difficulty  users  had  with  the  interface  to  the  haptic 
communication.  Most  users  found  the  haptic  communication 
hard  to  invoke,  and  this  is  an  area  that  requires  substantial 
improvement  before  further  trials  take  place. 

Figure  4  illustrates  the  results  of  presence  questionnaire.  In 
keeping  with  the  observation  that  the  haptic  communication 
may  increase  a  user’s  sense  of  presence,  the  Haptic  condition 
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Figure  3.  Results  from  TLX  Questionnaire 


yielded  significantly  greater  subjective  ratings  of  spatial 
presence  (p<0.05).  Also,  supporting  the  observation  that  users 
found  the  haptic  communication  appealing,  it  achieved 
significantly  higher  ratings  than  the  Visual  condition  in  the 
engagement  category  (p<0.05).  Finally,  the  Haptic  condition 
was  also  rated  significantly  higher  on  the  naturalness  factor 
(p<0.05).  'Fliis  factor  attempts  to  measure  how  interacting 
with  the  system  compares  with  interactions  and  experiences  in 
the  real  world. 

The  results  from  the  usability  questionnaire  arc  presented 
in  figure  5.  Overall  usability  was  significantly  improved  in  the 
Haptic  condition  (p<0.05)  as  were  the.  individual  factors  of 
system  usefulness  (pcO.OI)  and  interface  quality  (p<0.05). 

Results  from  the  custom  questionnaire  are  shown  in  figure 
6.  While  this  questionnaire  was  developed  simply  for  this 
experiment,  and  as  such  little  trust  can  be.  placed  in  the 
validity  of  the  data  that  it  produces,  it  is  promising  to  note  the 
unanimously  rated  superiority  of  the  Haptic  condition. 

5.  Discussion 

The  information  gained  from  the  questionnaires  supports 
that  gained  from  the  observation.  On  the  negative  side,  users 
experienced  problems  with  the  interface  to  the  haptic 
communication,  and  there  appears  to  be  substantial  variation 
in  the  adoption  of  the  communication,  probably  due  to  the 
personal  nature  of  the  sense  of  touch.  On  the  other  hand,  the 
majority  of  the  participants  appeared  to  find  the  haptic 
communication  engaging  and  used  it  frequently.  It  also 
significantly  increased  subjective  ratings  of  presence, 
improved  usability  and  appeared  to  facilitate  collaboration. 

The  positive  results  of  this  experiment  suggest  future 
avenues  of  research.  The  gesture  effect  appeared  useful  when 
indicating  complex  information,  for  instance  gesturing  to 
encompass  a  variety  of  objects,  or  a  complex  shape.  We 
suggest  that  an  evaluation  of  this  technique  in  such  situations 
would  yield  interesting  results.  The  locate  effect  was  well 
received  by  subjects  and  it  may  be  interesting  to  compare  this 
navigation  and  coordination  technique  to  other,  possibly 
visual,  aids.  Finally,  a  direct  comparison  of  the  effect  of  the 
proximity  and  push  effects  on  subjective  ratings  of  presence 
might  also  prove  interesting. 

In  conclusion,  this  work  has  allowed  us  to  assess  the 
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Figure  4.  Results  from  1TC  Presence  questionnaire 
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Figure  5.  Results  From  OS  Usability  Questionnaire 

suitability  of  this  novel  baptic  communication,  and  to  attempt 
to  determine  future  research  possibilities  that  might  stein  from 
it.  We  have  generated  several  hypotheses  for  future  work, 
and,  more  importantly,  found  that  the  majority  of  users  regard 
the  communication  as  appealing  and,  subjectively  at  least, 
gain  significant  benefits  from  it. 
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effectively  with  the  other  user 

2.  It  was  easy  to  find  the  other  user 
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effective 

4.  1  worked  alone 
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6.  Communicating  with  the  other  user  was  simple 

7.  1  often  found  it  hard  to  locate  the  other  user 

8.  I  was  not  aware  of  the  activities  of  the  other  user 

9.  The  system  did  not  support  my  desire  to  communicate 

10.  The  software  made  it  easy  to  work  as  a  team 

Data  has  been  adjusted  so  that  higher  values  indicate  higher 
levels  of  cooperation  for  all  questionnaire  items 
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Abstract 

A  cooperative  shared  haptic  virtual  environment, 
where  the  users  can  kinesthetically  interact  and 
simultaneously  feel  each  other,  is  beneficial  for  many  VR 
simulations.  We  have  implemented  a  tele-handshake 
system  that  enables  the  participants  to  shake  hands  over 
a  network  and  feel  each  other’s  pushing  concurrently.  A 
client-server  architecture  has  been  used  with  a  specific 
implementation  to  meet  the  requirements  of  the  haptic 
device.  The  users  were  able  to  feel  each  other 
simultaneously  and  shake  their  hands  in  an  intuitive  way. 
An  objective  evaluation  based  on  force  feedback  was 
conducted.  The  results  showed  that  the  force  feeling 
induced  at  the  remote  site  was  very  close  to  that  felt  at 
the  remote  site.  Also,  a  subjective  evaluation  based  on  a 
rating  questionnaire  is  described.  The  results  prove  that 
the  feeling  was  instant  without  any  perceptible  delay. 


1.  Introduction 

The  ability  to  feel  objects  in  a  shared  virtual 
environment  simultaneously  can  markedly  enhance  the 
effectiveness  of  many  VR  applications.  Haptic 
cooperative  virtual  environments,  where  the  users  can 
simultaneously  manipulate  and  haptically  feel  the  same 
object,  is  potentially  beneficial  and  in  some  cases  could 
be  indispensable.  It  could  be  useful  for  training  as  in,  for 
example,  virtual  surgical  simulators  where  the  team 
performs  on  the  same  real  patient;  all  the  other  members 
perceive  each  team  member  interaction  with  the  patient 
directly,  when  dealing  with  the  patient’s  tissue,  or 
indirectly  because  of  the  collisions  in  the  limited 
workspace.  Virtual  surgical  simulators  offer  the 
possibility  of  training  surgeons  without  risking  casualties 
[1]  [2]  [3].  However,  the  interaction  feeling  and  its  effect 
should  be  provided  to  make  sure  that  the  perceptual 
experience  in  the  virtual  world  corresponds  to  that  in  the 
real  world,  otherwise  the  training  would  be  useless,  if  not 


dangerous.  Another  use  of  a  haptically  cooperative 
environment  is  in  entertainment,  which  would  allow  the 
participants  to  kinesthetically  interact  with  each  other. 
This  adds  a  new  dimension  of  enjoyment  and  brings  us 
one  step  closer  to  more  realistic  interactivity.  Moreover, 
there  will  be  a  great  benefit  from  such  kinesthetic 
interaction  in  some  sports  training  systems,  especially  the 
kind  of  multi-players  sports  that  include  direct  contact 
between  the  players  such  as  boxing,  sumo  wrestling,  and 
football.  The  resulting  virtual  training  system  has  the 
potential  to  be  more  realistic  and  efficient.  In  addition, 
there  are  many  advantages  of  multi-hand  manipulation 
that  can  be  realized  from  daily  life.  More  precise 
manipulation  can  be  achieved  using  both  hands  [4],  Also, 
the  manipulation  with  both  hands  is  more  efficient  than 
one  hand  as  has  been  mentioned  by  some  ergonomic 
studies  [5]. 

Buttolo  et  al.  [6]  proposed  an  architecture  for  shared 
haptic  virtual  environments  where  they  pointed  out  the 
difference  between  collaborative  and  cooperative  virtual 
environments.  The  collaborative  environment  is  a  sharing 
environment,  in  which  the  users  take  turns  in 
manipulating  the  virtual  object.  Meanwhile,  the 
cooperative  environment  is  an  interacting  one,  in  which 
the  users  can  simultaneously  modify  the  same  virtual 
object.  According  to  these  definitions  most  of  the 
cooperative  haptic  environments  that  have  been  proposed 
in  the  last  few  years  fall  under  the  banner  of  collaborative 
haptic  environments,  as  all  of  them  cannot  support 
simultaneous  kinesthetic  interaction  between  the 
participants.  Takemura  and  Kishino  [7]  have  built  a 
cooperative  environment  using  a  virtual  workspace  by 
combining  a  head  tracking  display  and  a  data  glove. 
However,  the  users  are  not  allowed  to  simultaneously 
alter  the  status  of  the  virtual  world.  In  other  words,  they 
cannot  grasp  the  same  object  and  manipulate  it  at  the 
same  time.  They  used  what  they  called  mutual  exclusion 
to  avoid  simultaneous  manipulation  of  the  same  object.  In 
the  shared  virtual  environment  proposed  by  Buttolo  et  al. 
[8],  one  user  at  time,  the  active  operator,  is  allowed  to 
modify  and  feel  the  force  from  the  environment. 
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Meanwhile,  the  other  users  watch  the  interaction  hut 
cannot  modify  or  touch  the  object. 

Kinesthctically  linking  remote  users  acting  on  the  same 
object  is  a  great  challenge  as  any  latency  in  the  haptic 
rendering  loop  could  bring  the  whole  system  to  instability. 
The  delay  should  not  be  more  than  5-10  ms  to  accomplish 
a  stable  interaction  with  a  stiff  virtual  object. 
Handshaking  is  a  good  example  of  .simultaneous 
kinesthetic  interaction,  therefore,  it  is  an  important  task  to 
be  implemented  and  tested.  It  represents  a  highly 
demanding  task  for  concurrent  interaction  and  tpiick 
instant  feeling.  As  a  result,  achieving  a  rich,  instant  and 
realistic  haptic  sensation  during  tcle-htuidshakc  would  be 
an  indicator  for  the  possibility  of  implementing  a  high- 
quality  cooperative  haptic  virtual  environment..  Another 
reason  could  be  a  social  one  related  to  our  life 
communication  protocol  where  every  greeting,  farewell, 
meeting  or  agreement  starts  and  ends  with  handshake. 
Therefore,  representing  a  handshake  in  any  haptic 
cooperative  virtual  environment  will  encourage  a  more 
intuitive  and  natural  interaction. 

This  paper  describes  a  tele-handshake  system  with 
haptic  sensation  over  the  Internet.  An  implementation  of  a 
shared  cooperative  haptic  virtual  environment  where  the 
users,  at  distant  sites,  can  shake,  hands  and  feel  each  other 
simultaneously.  An  architecture  for  kinesthetic 
cooperative  haptic  interaction  in  a  shared  virtual 
environment  is  proposed.  An  experiment  and  evaluation 
of  the  proposed  system  arc  described. 

2.  The  experimental  system 

The  system  consists  of  two  connected  stations  in 
client-server  mode.  Each  machine  is  connected  to  the 
haptic  device  by  which  the  users  interact  inside  the  virtual 
space,  and  the  graphic  device  to  display  the  virtual 
environment  to  the  users  is  as  shown  in  Figure  1.  The 
users  from  different  locations  come  to  meet  each  other 
inside  the  virtual  meeting  space,  which  is  a  3D  spacious 
room  with  two  graphic  arms  representing  each  user’s  hand 
as  can  be  seen  from  Figure  2.  PHANloM  from  SensAblc 
Technologies,  Inc.  was  used  as  the  haptic  device  to 
represent  the  force  feedback  to  the  subjects  19].  I  be 
participants  were  asked  to  enter  the  virtual  meeting  space 
in  which  they  can  observe  each  other's  movement,  then  to 


shake  hands  as  naturally  as  they  usually  do  in  real  life. 
The  force  feedback  was  acquired  from  each  site  to 
evaluate  the  haptic  sensation  provided  by  the  local  and  the 
remote  location.  Figures  3  and  4  show  a  snapshot  of  the 
experimental  environment  at  each  site  during  a  real  tele- 
handshake  session. 


Figure  1.  Diagram  of  the  tele-handshake 
system 


Figure  2.  The  virtual  environment  (virtual 
meeting  space) 
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Figure  3.  The  experimental  environment 
at  site  1 


Figure  4.  The  experimental  environment 
at  site  2 


3.  The  tele-handshake  system  architecture 
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Figure  5.  The  system  architecture 
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The  architecture  can  be  described  as  a  client-server 
structure  as  can  be  seen  from  figure  5.  The  system  that 
has  been  used  to  implement  the  tele  handshake  consists  of 
two  machines  at  different  locations. 

The  server  program  and  client  program  arc  running 
concurrently  at  each  site  at  the  same  time.  The  server 
program  is  responsible  for  initiating  the  haptic  device  and 
passing  force  and  position  information  to  the  client  side. 
The  client  program  is  the  application  code,  which  initiates 
and  draws  the  graphic  and  haptic  scenes,  then  establishes 
the  communication  channel  with  the  server  to  receive  the 
data  from  the  haptic  device  using  callback  mechanism. 
This  mechanism  passes  only  the  new  state  information  for 
the  nodes  in  the  haptic  scene  from  the  server  side  to  the 
client  side.  This  technique  saves  traversing  all  the  nodes 
in  the  scene  checking  Cor  any  changes.  After  that,  the 
haptic  and  graphic  scenes  are  updated  according  to  the 
new  state  information  that  was  received  from  the  server 
side. 

4.  Results  and  evaluations 

4.1  Objective  evaluation 

Linking  haptic  devices  over  the  network  is  a  great 
challenge  because  of  the  small  delay  requirements  and  the 
high  update  rate  required  for  haptic  rendering.  A  1000  Hz 
update  rate  was  suggested  by  Buttolo  and  his  colleagues 
to  create  the  feeling  of  stiff  surfaces  and  to  prevent 
unwanted  sudden  forces  [10].  Using  the  above 
architecture  enabled  us  to  meet  these  demands  over  the 
network.  By  keeping  the  haptic  process  running  on  the 
server  side  at  a  high  update  rate.  Meanwhile,  the  graphic 
process  was  running  at  the  client  side  and  exchanging  the 
necessary  information  with  the  haptic  process,  which  are 
usually  forces  and  positions  that  can  be  transferred 
efficiently  through  the  current  Internet  network. 

To  evaluate  the  system  we  compare  between  the  force 
feedback  provided  to  the  local  client,  who  is  connected  to 
his  haptic  device  on  the  local  machine,  and  the  forces 
provided  to  the  remote  client  that  is  connected  to  the 
haptic  device  on  the  remote  machine.  Therefore,  it  was 
sufficient  to  obtain  the  forces  from  the  local  side  and 
those  from  the  remote  side,  and  compare  them  to  quantity 
the  haptic  sensation  provided  at  each  site. 

The  forces  in  (he  x  and  z  directions  from  the  remote 
site  could  provide  up  to  87%  of  the  forces  at  the  local  site. 
Meanwhile,  the  forces  in  the  y  direction  were  up  to  92% 
of  that  at  the  local  site  as  can  he  seen  from  Figure  6.  The 
results  reflect  the  nature  of  the  handshake  task  since  we 
usually  intend  to  shake  the  partner's  hand  up  and  down  in 
the  y  direction. 


t  /  tocnf  F *  /timuie  ry  loc.il  fy-mnioto  FV-loeol  F/  remoto 


Force  Category 


Figure  6.  The  local  artel  remote  force  feedback 
evaluation 

4.2  Subjective  evaluation 

Evaluating  human  interaction  with  any  VR  simulation, 
in  general,  and  haptic  VR  simulation  in  particular  is  a 
complex  task  since  there  arc  many  factors  that  cannot  he 
considered.  We  used  a  simple  subjective  evaluation  using 
a  rating  questionnaire  to  show  to  what  extent  the  system 
could  deliver  a  convincing  feeling  and  what  kind  of 
features  have  been  conveyed  to  the  users.  Ten  subjects 
were  asked  to  fill  out  the  questionnaire  shown  in  Table  1. 


Table  1.  Questionnaire  rating  the  feel  of  the 
simulation 


Convincing 

feeling 

1-2—3  4-5- 6-7 

Unconvincing 

Intuitive 

manipulation 

]. .2-3-4  5  6-7 

Unnatural 

manipulation 

Satisfied  with  the 
interface 

1-2-3-4-5-6-7 

Dissatisfied 

Instant  feeling  of 
forces 

L -2-3 -4 -5-6-7 

Slow  (delayed) 

Haptic  sense 
consistent  with 
the  visual  sense 

1-2-3  4-5  6  -7 

Inconsistence 

As  can  he  seen  from  Figure  7,  the  subjects  thought  that 
the  system  could  deliver  an  instant  feeling  without  any 
noticeable  delay.  Moreover,  the  haptic  sensation  was 
consistent  witli  the  visual  sensation,  up  to  95%.  The 
interface  was  not  fully  satisfactory  because  the  Phantom  is 
more  like  grabbing  a  pen  than  whole-hand  grasp  as  in  real 
handshake  situation;  only  70%  of  satisfaction  with  the 
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interface  has  been  expressed.  This  might  explain  why  the 
subjects  were  partially  convinced  with  the  feeling  and  the 
intuitiveness  of  manipulation.  They  were  up  to  62% 
convinced  with  the  stimulated  feeling  compared  with  the 
real-life  handshake’s  feeling. 


loo- 


Figure  7.  Subjective  evaluation  according  to 
the  rating  questionnaire 

5.  Conclusions 

In  this  paper  we  described  a  system  for  tele  handshake 
over  the  Internet.  A  client-server  architecture  was  used  to 
connect  the  users  with  their  haptic  device.  Moreover,  a 
callback  mechanism  was  used  to  communicate  the  haptic 
data  between  the  server  and  their  clients  As  a  result,  there 
is  a  reduction  in  traffic  since  only  relevant  changes  are 
communicated. 

The  participants  at  different  locations  could  enter  the 
virtual  space  meeting  and  shake  hands  and  teel  each  other 
instantly.  The  system,  at  the  remote  side,  could  provide  up 
to  92%  of  the  force  feeling  induced  at  the  local  side, 
which  makes  using  the  haptic  devices  over  the  network 
applicable  and  efficient.  The  feeling,  as  has  been 
described  by  the  subjects,  was  instant  without  perceptible 
delay. 

The  subjects  expressed  their  desire  to  have  the  same 
experience  but  with  the  ability  to  use  the  whole  hand, 
because  that  would  deliver  a  more  satisfactory  and 
realistic  feeling.  Nevertheless,  this  would  open  up  a  new 
dimension  of  direct  human  contact  and  interaction  over 
the  network.  The  ability  to  touch  and  feel  others  far  away 
would  definitely  start  a  new  era  in  human  communication 
and  would  give  us  access  to  the  development  of  new 
applications  over  the  Internet  that  previously  only  existed 
in  our  imagination. 
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Abstract 


operating  at  the  rate  of  IKlTz,  from  which  tire  data  is 
extracted  arid  sent  to  the  network  at  a  desired  lower  rate. 


This  paper  focuses  on  the  problems  of  networked 
gestures.  The  tele-gesture  is  the  achievement  of  a  remote 
gesture.  To  reach  a  good  season  perception  of  the  distant 
scene,  it  is  necessary  to  create  a  specific  flow  for  the 
haptic  data:  position,  orientations,  and  feedback  forces. 
This  flow  presents  intrinsically  a  bi-directional 
characteristic.  Its  quality  is  ext  remedy  dependent  on  the 
latency  of  the  intermediary  telecom  network.  In  this  paper 
we  will  present  the  results  of  our  experiments  on  the  tele- 
gesture. 


1.  Introduction 

The  tele-gesture  is  the  achievement  of  a  remote 
gesture,  i.e.  to  touch  a  distant  object,  that  is  real  or  virtual, 
for  example,  a  human  body,  and  to  feel  its  mechanical 
characteristics,  sensitivity  specific  to  the  bones,  muscles, 
tendons  and  joints  which  give  information  about  its  static, 
balance  and  the  displacement  of  tire  body  in  space,  see 
1 3 1,  [5],  To  reach  a  good  sensory  perception  of  the  distant 
scene,  it  is  necessary  to  create  a  specific  flow  for  the 
haptic  data:  position,  orientations,  and  feedback  forces. 
This  flow  presents  intrinsically  a  bi-directional 
characteristic.  Its  quality  is  extremely  dependent  on  the 
latency  of  the  intermediary  telecom  network.  To  ensure  an 
encoding  of  the  gesture,  it  is  advisable  to  use  a  virtual  3D 
object,  of  which  it  is  necessary  to  build  a  coherent  visual 
and  haptic  representation.  Each  scene  is  duplicated  on  the 
two  connected  sites:  on  the  local  machine  and  on  the 
distant  machine.  The  haptic  device  frequency  is  too  high, 
typically  I  KHz,  so  it  would  not  be  possible  to  go  forwards 
and  backwards  through  the  network  at  such  a  rate!  On 
each  machine,  one  must  thus  establish  a  local  loop 


A  touch  robot 
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-*  flic  distant  and 
distributed  touch 


Figure  1.  Distant  Touch  and  distributed  touch 

A  SensAble  PIIANToM  haptic  robot  is  used.  This 
device  has  6  degrees  of  freedom  and  renders  a  three- 
dimensional  force  information.  It  can  track  the  position 
and  orientation  of  the  tool  within  a  workspace  of  16  cm 
wide,  13  cm  high  and  13  cm  deep.  The  maximum  exerted 
force  is  6.4  Newton.  We  employ  a  second  PHANToM  as 
the  distant  system,  i.e.:  the  slave  system,  sec  Figure  1. 
There  are  three  main  kinds  of  experiments  of  the  tele 
gesture: 

•Distributed  touch  like  the  example  of  the  fencing, 

•Touch  of  ballistic  object,  whose  trajectory  is  foreseeable, 
like  tennis  games, 

•Static  touch  of  object,  the  telc-cchography  is  an  example 
of  static  touch,  with  a  pressure  sensor  on  the  patient 
abdomen. 

2.  Local  model  and  distant  model 

The  3D  geometrical  model  of  the  distant  scene  is 
coded  with  polygonal  objects.  The  grid  is  deformed  under 
the  gesture.  The  fine  touch  is  the  notion  of  touching  free 
forms.  Two  approaches  were  studied  in  our  projects: 
touch  of  the  facets  and  touch  of  the  nodes. 
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Touch  of  the  facets:  it  is  the  calculation  of  the  position 
of  a  point  that  is  named  the  haptic  point1  ,  compared  to  a 
selection  of  facets  to  touch  -  when  the  scalar  product 
between  the  normal  to  the  touched  lace  and  the  vector 
going  from  the  haptic  point  to  its  projection  on  the  same 
face  changes  sign,  the  haptic  point  crosses  the  facet.  The 
force  will  then  be  calculated  as  F  =  -  K  X.  (K  is  the 
stiffness  of  the  virtual  object  at  the  considered  face  and 
may  be  f  ixed  or  dynamically  adjusted  and  X  outdistance 
the  haptic  point  to  the  facet).  For  this  approach  the 
continuity  of  the  force  is  not  induced  and  the  processing 
of  smoothing  is  necessary.  Indeed,  the  force  must  he 
continuous  when  the  user  makes  the  handle  slide  along  the 
surface  of  die  object  so  that  the  haptic  point  projection 
changes  facet. 

Touch  of  die  nodes:  it  is  the  calculation  of  the  haptic 
point’s  position  compared  to  a  selection  of  points:  this 
approach  considers  a  sphere  of  radius  R,  the  object  nodes, 
the  vertices!  |,  and  the  haptic  point.  When  the  sphere, 
attached  virtually  to  the  haptic  point,  contains  polygonal 
object  nodes,  then  a  force  is  returned  proportionally  to  the 
number  of  touched  points,  and  the  nodes  move  under  the 
gesture.  A  zone  of  forces  is  produced  to  allow  a  feeling  as 
realistic  as  possible  of  the  3D  scene.  'I  lie  effort  feedback 
is  calculated  starling  from  a  physical  model  of  the  virtual 
3D  form  that  is  touched. 

Table  1.  Local  model:  the  qualitative  results  of  experiments. 
These  tests  have  been  performed,  works  in  local  loop  i.c.  without 
information  coming  from  the  network  concerning  the  distant 
slave  workstation. 


Touch 

Feel 

Haptic 

rendering 

Conclusion 

on  a  set  of 
vertices 

Discontinuous 

Correct 

Not  realistic  for 
the  fine  gesture. 

on  one  polygon 

Comfortable. 

Good 

Realistic 

on  one  polygon 
and  on  its 
vicinity 

Natural, 

comfortable 

Very 

good 

Realistic. 

Precise 

The  material  poinMs  heavy,  clastic  and  viscous.  1  he 
state  of  the  material  point  is  governed  by  mechanical 
equations.  The  3D  scene  objects  are  flexible,  deformable 
and  their  mechanical  characteristics,  viscosity,  rigidity, 
and  mass  can  be  adjusted.  We  tried  out  three  ways  of 
feeling  this  modelling  dependent  on  the  contact  with  a 
polygonal  object.  The  first  is  a  calculation  of  force 
feedback  based  on  the  nodes  of  the  geometrical  3D  model. 
The  second  allows  a  force  feedback  according  to  the 
nearest  polygon  to  the  haptic  point  position.  The  third 
comes  from  the  second,  and  allows  an  effort  feedback 


*  The  haptic  point  is  the  extremity  of  the  phantom  arm  which  is  the 
point  of  application  of  the  force,  in  the  virtual  world. 


according  to  the  polygon  closer  to  the  haptic  point,  but 
also  according  to  its  vicinity.  We  note  the  qualitative 
results  of  our  experiments  in  the  'fable  1. 

Gesture  sequences  can  also  be  recorded  to  make  a 
simulation,  or  to  replay  these  sequences.  We  can  build 
gesture  cards.  We  must  take  into  account  the 
physiological  constraints  of  the  human  gesture  for  the  tele- 
gesture  to  be  possible.  The  haptic  loop  is  fed  at  a 
frequency  of  Ik! lx.  because  below  this  frequency  the 
gesture  perception  would  be  of  bad  quality,  i.e.: 
unpleasant  vibrations  would  be  perceived  in  the  hand. 


Master  system 
X"(positions.  anplcs) 


Virtual  Model 


K  stiffness 

•a/naa-- 

K*  stiffness 


Slave  system 
Y-(posil  ions,  angles) 
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Figure  2.  Distant  control  of  the  robot  in  position 


Wc  use  the  local  model  of  the  distant  scene  to  calculate 
the  force  feedback  at  this  rate.  The  network  will  feed  this 
local  model,  see  figure  1.  The  network  is  materialized  as  a 
spring.  It  appears  necessary  to  have  a  local  vision:  a 
geometrical  model,  mechanical  3D  front  what  occurs  to 
the  distant  one.  The  local  model  is  a  precise  model  of 
what  occurs  locally.  This  model  interacts  at  the  frequency 
of  the  robot  with  the  local  robot..  The  distant  model 
summarizes  and  memorizes  for  the  local  model  all  that  is 
held  within  the  distant  one.  The  distant  model  is  fed  by  the 
network  messages.  The  distant  model  feeds  the  local 
model  directly. 

Contrary  to  the  wave  transform  approach  [2],  we 
exchange  haptic  data  directly  in  space  of  the  temporal 
signals.  Our  approach  is  based  on  distant  virtual 
mechanical  model. 

3.  Evaluation  of  haptic  flow 

Through  the  network,  the  data  are  encapsulated  in 
messages.  In  general,  the  messages  contain:  the  positions 
X,  Y,  Z,  the  angles  a,  P,  y,  and  the  date.  The  size  of  the 
messages  is  dependent  on  the  workspace  that  is  to  be 
encoded.  The  working  area  for  the  handle,  the  workspace 
is  160x130x130  and  the  precision  is  of  0.02mm,  for  a 
SensAble  TlIANToM  type  desktop. 
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Equation  1  Evaluation  of  number  of  points  to  encode 

iiiaxiiiiul  bandit*  ambitus  3 

- — - -  maximum  number  of  points  to  lie  coded  r  O' 


precision 

where  t>^  is  the  working  area 


By  applying  Equation  1,  one  obtains  the  maximum 
number  of  points  to  be  coded  :  8000  points.  Then,  the 
positions  can  be  coded  on  13  bits  as  2 12  =4096  and 
213  =8192. 

Equation  2  Message  size  limits 

network  flow 

- - : - : —  -  iiuixiimim  siyc  oF  message 

sampling  jMTiod 

The  message-sampling  period  is  one  of  the  parameters 
that  regulates  experimentation,  see  Table  2.  The  network 
flow  and  the  size  limits  of  messages  are  correlated.  For 
example,  the  ISDN  network  flow  is  64  kbits  per  second.  If 
the  sampling  period  is  10ms  then  the  maximal  size  of  a 
message  is  640  bits,  sec  Equation  2. 

The  latency  of  the  networks  is:  30  ms  for  ISDN,  100  to 
200ms  for  ADSL2,  around  10ms  for  the  local  area  network 
LAN.  The  ADSL  network  is  not  recommended  for  tele- 
gesture,  whereas  the  ISDN  is  adequate. 


4.  The  predictive  coding  of  0,  1  and  hybrid 
order 

Messages  contain  at  least  a  stamp  and  haptic  point 
position  coordinates  through  a  network  supposed  "ideal". 
It  means  that  the  only  disturbance  applied  to  the  data  is  a 
delay  corresponding  to  the  duration  of  the  data’s  route.  We 
use  the  local  model  of  the  distant  patient  to  calculate  the 
force  feedback  at  a  1  kHz.  rate.  The  network  will  feed  the 
expert  master  interface  device  with  real  measurements  at  a 
lower  rate.  So  we  need  to  use  a  predictive  algorithm  to 
calculate  the  signal  to  send  to  the  robot. 


Synopsis  of  Mt-fii'Slui't 


R  :  received 
D  :  distant 
L  :  local 

8  :  delay  of  the  network 
t  :  temporal  index  of  the  local  model 
T  :  temporal  index  of  the  distant  model 

Figure  3.  Synopsis  of  tele  gesture 


^  Asymmetric  Digital  Subscriber  I. inc 


The  network  is  like  a  delay  line.  It  results: 
X§  =  X \J  y  •  Wc  want  to  evaluate  xp  and  wc  dispose  of 

the  following  information: 

. ■*£-(*•-«)  the  serie  of  position 


vectors  which  transit  through  the  network,  and  n  the 
buffer’s  length  in  which  the  received  data  is  written.  The 
real  time  imposes  the  implementation  of  the  distant 
model,  see  Figure  2.  The  distant  model  is  a  reproduction 
closer  as  possible  to  the  local  model,  knowing  that  we 
have  only  the  past  data  of  the  local  model.  This  past  data 
arc  separated  from  the  present  of  our  distant  model  by  the 
incompressible  network’s  latency,  i.e.  the  route’s  duration 
of  the  information.  We  are  going  to  look  for  creating  a 
relation  of  the  type: 


for  each 


of  three  position  vectors. 

For  a  predictive  coding  of  order  0,  there  is  a  brutal 
modification  of  the  distant  model  state  when  the  message 
arrives,  see  Figure  4.  The  variable  state  representing  the 
distant  model  is  equal  to  the  just-received  data.  In  this 
case:  the  function  \\i  is  the  simplest,  following  this 
relation:  X  p  -  From  the  arrival  of  the  message 

we  modify  brutally  the  distant  model’s  state,  we  equalize 
the  state’s  variable  representing  the  distant  model  to  the 
just  received  data. 

In  a  continuous  haptic  flow  transmission  case,  a  secure 
network  (without  loss)  case  and  a  constant  transmission 
speed  case:  the  distant  model  determines  the  signal,  which 
is  in  fact  the  position  of  the  haptic  point  and  is  equal  to  the 
signal  of  local  model’s  signal  but  displaced  in  time. 


”  Qiginal  signal 


For  a  predictive  coding  of  order  1,  information  on  the 
slope  of  the  original  signal  is  used  to  predict  the  signal 
between  two  messages  from  the  network,  see  Figure  5.  In 
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that  case  the  function  \|/  is  tiffin:  'P(a)-- u*x  +  b  .  The 
following  relation  is  deduced:  x  t  ~  A  p  +  $  *  E  > 
with  P[f\: ,  l\, ,  /J- )  the  slope  vector  composed  of  the 

slope  calculated  on  axis  X,  of  the  slope  calculated  on  axis 
Y,  and  of  the  slope  calculated  on  the  axis  Z.  The  slope  is 
estimated  in  the  following  way: 


By  proceeding  in  this  way,  there  is  a  weak  noise  of 
quantification  on  the  signal  representing  the  position  of 
the  haptic  point;  this  noise  is  amplified  in  our  construction 
of  the  predicted  signal  because  it  is  multiplied  by  the 
factor  5  corresponding  to  the  network's  latency.  It  is 
important  to  smooth  the  received  signal  to  decrease  the 
disturbance  generated  by  this  quantification’s  noise.  We 
observe  that  the  frequency  of  this  noise  was  included 
between  150  and  200  1-Iz.  We  smooth  the  slope  by  using  a 
temporal  window  of  length  n-10  ms. 

The  now  slope’s  calculation  is. 
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This  signal  is  belter  than  the  one  predicted  with  the 
order  0,  especially  in  the  straight  lines.  However,  it  has 
poor  results  during  the  breaks  in  the  slope  of  the  original 
signal. 


( Mginal  signal 


For  a  predictive  coding  of  hybrid  order,  the  calculated 
signal  is  the  half  sum  of  the  signal  obtained  by  a 
predictive  coding  of  order  0  and  the  one  obtained  by  a 
predictive  coding  of  order  1,  sec  Figure  6. 


Signal  oblainat  by  n  predictive  . Original  signal 

toiling  of  outer  0  - Sampling  signal  received  from 


Signal  obtnmedby  a  predictive 


network 


Figure  (y.  Predictive  coding  of  hybrid  order 


We  have  smoothed  the  predicted  signal  as  this  last  one 
results  from  an  affin  function  of  the  input  signal.  The 
input  signal  arises  from  the  digitalization  of  an  analogical 
signal  representing  the  hand  movement,  which  acts  on  the 
robot  arm.  As  the  hand  of  the  user  vibrates  slightly  it 
appears  teeth  in  the  digitized  signal,  sec  the  Figure  7,  it  is 
observed  in  a  tenth  of  millimeter  scale.  The  smoothing 
method  consists  in  using  a  linear  regression. 
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Figure  7.  Example  of  the  smoothing  method 
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4.  Tele  gesture  experiments 

These  experiments  were  performed  using  two 
PHANToM  systems  communicating  through  the  network 
[l].  Each  device  sends  its  position  and  receives  the 
position  of  the  other.  We  tested  the  initialisation  and  shut 
down  of  the  haptic  session,  the  transmission  of  die 
position  to  the  distant  device,  and  the  calculation  of  the 
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efforts  at  lkhz  with  a  data  refreshment  at  the  reception 
with  various  rates  ranging  from  10  to  500ms.  The  first 
experimental  results  are  reported  in  Table  2. 

The  case  of  the  synchronous  network  (as  ISDN)  allows 
an  interaction  of  the  real  gestures  regularly.  The  up-to- 
date  handing-over  of  the  extrapolator  of  gesture  is  thus 
carried  out  regularly,  according  to  a  fixed  period, 
dependent  on  the  network. 

The  first  experiments  make  it  possible  to  test  the  first 
exchanges  of  gesture  agility  by  the  intermediary  of  the 
ISDN3  network: 

•Initialization  of  the  haptic  scene, 

•Stopping  the  haptic  scene, 

•Sending  the  position  of  the  local  robot, 

•Reception  of  the  distant  one  at  various  rates:  500,  100,  10 
ms, 

•Calculating  the  efforts  at  a  frequency  of  lkhz  with  a 
distant  robot  data  refreshment  at  reception. 

The  first  experiments  are  carried  out  with  an  ordering 
of  position  control  on  both  sides  of  the  network,  and  it 
was  done  the  first  time  for  tele-echography  [4].  The 
predictive  coding  is  of  order  1 ,  and  the  signal  is  smoothed. 
We  use  a  second  PHANToM  to  simulate  the  distant 
system,  ie:  the  slave  system.  Each  robot  sends  its  position 
and  receives  the  position  of  the  distant  robot,  see  figure  2. 


Master  system 


X=<positions,  angles) 

- - - ► 

Y=(positions,  angles) 


Slave  system 


Virtual  model  Network  Wood  mannequin  strato-conception 

Figure  8.  Position  control  on  both  sides  of  the  network. 


The  mechanical  model  used  in  the  first  experiments  is 
a  link  between  the  local  model  and  the  local  perception  of 
the  distant  model.  At  this  point,  the  best  way  of  sampling 
the  messages  must  be  found,  the  latency  time  due  to  the 
network,  the  robot  1kHz  period  of  sampling,  and  to  the 
optimal  period  of  sampling  for  the  gesture  signal.  We 
redefine  the  extrapolation  algorithms  of  the  positions  and 
forces,  and  we  test  the  predictive  coding  of  0,  1  and  hybrid 
order  of  the  distant  mechanical  models.  Specific  filtering 
of  emissions  and  receptions  of  the  gesture  data  is 
introduced.  At  the  emission,  it  is  a  question  of  filtering  the 
natural  noise  made  by  the  hand.  At  the  reception,  it  is  a 
question  of  filtering  the  jitter  introduced  by  the  variable 
times  of  message  transport,  such  as  adding  a  low-pass 
filter  on  the  speed  of  the  gesture  signals. 


Figure  9.  The  tele-echography  command  station  -  a 
slow  gesture 


To  improve  gesture  ergonomics  of  the  expert  within 
the  tele-echography,  we  will  create  the  virtual  tangential 
forces,  which  increase  quickly  if  the  medical  expert 
moves  his  hand  too  quickly,  because  the  distant  robot  will 
have  its  own  speed  limits,  see  the  green  vectors  on  figure 
2.  The  medical  expert  feels  the  force  exerted  on  the 
patient  and  measured  with  a  sensor,  perpendicularly. 
Figure  3  shows  the  current  interface  of  the  haptic 
command  station  for  tele-echography.  The  green  segment 
materializes  the  intensity  of  the  force  perceived  by  the 
medical  expert. 


Table  2.  Transmission  experiments  using  the  ISDN  network: 


Row 

100Hz 

10Hz 

4Hz 

2  Hz 

Very  good 

Good 

Some 

vibrations  and 
oscillations 

Command 

ability 

Natural 

Restricted  in 
term  of  the 
gesture 
speed 

Difficult 

Very  difficult 

Conclusion 

Realistic 

Realistic  for 

slow 

gestures 

Need  to  set  up 

finer 

extrapolators 
of  gesture 

Need  to  set  up 

finer 

extrapolator 
and  stabilizer 
of  gesture 

The  case  of  the  asynchronous  network  (as  internet) 
allows  an  irregular  interaction  of  the  real  gestures.  The  up- 
to-date  handing-over  of  the  extrapolator  of  gesture  is  thus 
carried  out  irregularly,  according  to  a  non-fixed  period, 
dependent  on  the  network. 

The  fast  gestures,  such  as  the  sword  gesture,  require  a 
very  low  latency  time  of  the  intermediary  telecom 
network.  It  should  be  less  than  50  ms.  Figure  10  shows  the 
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example  of  the  swords,  a  "game"  of  gestured  agility.  The 
current  application  calculates  the  intersection  of  two  lines 
in  space,  i.e.  the  two  swords.  One  of  the  swords  is  brought 
under  control  of  the  local  robot;  the  local  player  handles 
it.  The  second  sword  is  the  local  model  from  what  occurs 
on  the  station  according  to  the  distant  player. 


Figure  10.  Sword  gesture-  a  fast  gesture 


that  the  gesture  can  be  transported  with  the  help  of  some 
ad  hoc  filtering,  by  ISDN  network,  or  on  IP.  The  transport 
of  the  gesture  data  via  the  network  allows  an  immersion 
even  more  complete  and  realistic  for  distant  manipulators. 
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This  game  uses  the  LAN4  network  with  UDP/IP5.  The 
client  side  has  to: 

•Initialize  the  client, 

•Create  a  socket, 

•Receive  messages  at  least  at  the  message-sampling  rate, 
•Send  messages  to  the  sending  message  client,  at  the 
message-sampling  rate  500,  100,  10  ms, 

The  server  side  has  to: 

•Initialize  the  server, 

•Create  a  socket, 

•Send  messages  at  the  message-sampling  rate  10  ms, 
•Receive  messages  at  least  at  the  message-sampling  rate, 
For  the  "game"  of  swords,  the  sampling  period  was 
10ms.  The  predictive  coding  was  of  order  1,  and  the 
signal  is  smooth.  The  IP  network  is  not  a  secure  network 
and  messages  can  be  lost.  It  is  also  asynchronous  network. 
In  our  experiment  the  latency  was  almost  less  than  10ms 
so,  the  results  and  the  feeling  were  realistic  and  natural. 

5.  Conclusion 

We  experimented  the  transmission  of  haptic  data  on  the 
ISDN  network  and  also  on  local  network  (LAN).  The 
results  of  our  experiments  are  very  encouraging  and  prove 
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Abstract 

Neuropathic  pain  patients  often  have  lemniscal-mediated 
symptoms  such  as  tactile  allodynia.  A  novel  method  and 
device  for  testing  lemniscal  pathway  integrity  is  described 
using  a  tactile  illusion  (Tl)  in  patients  with  neuropathic 
pain  resulting  from  peripheral  nervous  system  (PNS)  or 
central  nervous  system  (CNS)  injury.  An  unexpected 
enhanced  perception  of  the  Tl  was  found  in  some  PNS 
injured  pain  patients  with  mononeuropathies,  in  the 
majority  of  cervical  spinal  cord  injured  and  in  nearly  half 
of  brainstem  injured  central  pain  patients.  A  disinhibition 
of  lemniscal  pathway  mediated  Tl  was  found  in  these 
patients.  A  subset  of  thalamic  pain  patients  have  a  unique 
Tl  perseveration  or  persistent  after-sensation  despite 
reduced  Tl  perception  on  their  affected  side  compared  to 
their  normal  side.  The  results  of  these  studies  indicate  a 
unique  form  of  sensory  disinhibition  not  previously 
described. 

Introduction. 

Sensory  illusions  occur  in  the  tactile,  auditory  and  visual 
systems  [1].  The  tactile  illusion  (Tl)  of  movement  includes 
the  phi  phenomenon  and  is  defined  as  the  perceived 
movement  between  2  or  more  fixed  alternating  points  of 
vibration  on  the  skin  [2].  The  Tl  perception  is  dependent 
on  intact  peripheral  and  central  nervous  system  pathways 
including  lemniscal  pathways.  The  lemniscal  pathway  to 
the  cerebral  cortex  is  necessary  for  tactile  discrimination. 
Cortical  integration  of  peripheral  stimulation  is  needed  to 
form  the  Tl  of  perceived  movement  between  points  of 
vibration  that  form  symbols  such  as  numbers  "written"  on 
the  skin.  This  is  distinguished  from  graphesthesia,  defined 
as  the  ability  to  recognize  numbers  traced  lightly  on  the 
skin  with  a  stylus  or  pencil.  Both  lemniscal  (pressure, 
touch,  and  vibration)  and  reticular  (pain  and  temperature) 
neural  pathways  can  be  affected  by  a  disease  of  the 
peripheral  and  central  nervous  system.  This  is  due  to  the 
close  anatomic  proximity  of  these  pathways  so  that 
pathologic  injuries,  such  as  diabetic  neuropathies, 
traumatic  nerve  and  spinal  cord  injuries,  strokes  to  the 
brain,  etc.,  primarily  affecting  one  pathway  can  often 
affect  the  other.  Injury  to  a  single  peripheral  nerve  can 


result  in  a  perversion  of  the  tactile  sense,  called  allodynia, 
in  which  normally  innocuous  light  touch  (mediated  by 
lemniscal  pathways)  is  perceived  as  noxious  or  painful. 
Noxious  stimuli  are  normally  only  mediated  by  reticular 
pathways.  Disease  or  injury  to  the  central  nervous  system 
(CNS)  may  produce  similar  distortions  of  lemniscal  and 
reticular  pathway-mediated  sensations. 

In  these  studies  patients  with  CNS  and  peripheral 
nervous  system  (PNS)  injury  were  tested  using  a  device 
that  produced  the  Tl  of  numbers  “drawn”  on  the  skin 
across  several  dermatomes.  These  studies  are  described 
below  and  details  have  been  recently  published  [3,  4], 

Methods. 

In  the  2  studies  described,  patients  were  identified  from 
the  authors'  [GRG]  clinical  practice,  Mayo  Clinic, 
Scottsdale,  Arizona,  U.S.A.  An  institutional  review  board 
protocol  was  submitted  and  approved  for  these 
investigations,  and  informed  consent  was  obtained  from 
all  patients  and  controls.  A  room  that  was  free  of 
distractions  was  used  in  testing  all  subjects  in  both  studies. 
The  tactile  device  holds  solenoids  that  vibrate  and 
stimulate  discreet  cutaneous  points  and  produce  the 
illusion  of  "drawing"  complex  symbols  (i.e.  numbers)  on 
the  skin  where  the  device  is  applied  (see  Figure  1). 

In  the  first  study,  27  patients  with  unilateral  central  pain 
resulting  from  a  CNS  injury  to  the  cervical  spinal  cord, 
brain  stem,  lateral  thalamus,  or  suprathalamic  brain 
regions  were  studied  using  this  device.  The  device  is 
applied  to  the  volar  wrist  on  the  unaffected,  normal  side 
first  and  then  to  the  affected,  painful  side.  In  a  second 
study,  8  subjects  with  PNS  injury  were  tested  using  the 
same  study  procedures.  The  magnitude  of  the  learning 
effect  of  delivering  sequential  stimuli  was  determined  by 
testing  the  device  on  10  normal,  testing-naive  individuals. 
The  learning  effect,  L,  was  expressed  as  L=(T]/T2)xl00, 
where  Ti  is  the  total  number  of  attempts  to  identify  the  10 
random  digits  presented  on  the  first  body  side  tested  and 
T2  is  the  total  number  of  attempts  to  identify  the  10 
random  digits  presented  on  the  second  body  side  tested. 
The  first  side  was  tested  immediately  after  testing  the 
second  side. 
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Tactile  Illusion-Producing  Device 
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Figure  1.  The  tactile  illusion-producing  device 
with  solenoids  protruding  demonstrating  the 
number  "7".  The  frame  holds  18  solenoid  drivers 
with  vibrating  cores  in  an  array  measuring  27mm 
x  33mm  against  the  wrist.  The  sequence  of 
solenoid  protrusion  for  each  of  the  10  numbers 
delivered  can  be  found  in  reference  [3].  The  fixed 
settings  on  the  tactile  device  include,  5 
protrusions/solenoid  driver;  5  msec  interval 
between  protrusions;  10  msec  solenoid  on-time 
and  10  msec  solenoid  off-time 


Results.  In  measuring  the  learning  effect,  9  of  the  10 
normal  subjects  improved  in  that  they  required  fewer  trials 
to  identify  correctly  the  10  randomly  delivered  numbers 
on  the  second  side  tested.  The  median  learning  effect,  L 
was  1.23%.  This  constant,  L,  was  then  applied  to  the 
number  of  trials  on  the  affected  side  of  the  patients  with 
pain  from  CNS  and  PNS  injury  to  yield  the  Affected  Side, 
Adjust cd-for-lx'arning  va1ucs=L23.v,  where  ,v=the  number 
of  trials  on  the  affected  side. 


As  expected,  in  the  first  study,  8  of  10  patients  with 
thalamic  and  suprathalamic  central  pain  had  decreased  TI 
perception  on  the  affected  side.  Unexpectedly,  7  of  10 
patients  with  cervical  spinal  cord  central  pain,  and  3  of  7 
patients  with  brainstem  central  pain  had  better  TI 
perception  on  their  affected  side.  In  other  words,  fewer 
number  of  trials  were  needed  for  them  to  correctly  identify 
the  number  presentation  when  their  abnormal  side  was 
compared  to  the  normal  side.  The  enhancement  of  the 
perception  of  the  TI  in  the  majority  of  patients  with 
cervical  spinal  cord  central  lesions  and  in  nearly  half  of 
the  brainstem-injured  patients  was  unexpected.  Another 
observation  from  this  study  was  that  2  of  4  patients  with 
thalamic  injury  had  a  unique  phenomenon  of  TI 
perseveration,  despite  a  decreased  TI  perception  on  the 
affected  side.  This  perseverance  was  maintained  and  more 
than  one  number  could  he  felt  simultaneously  even  after 
subsequent  stimulation  to  that  body  part  was  discontinued 

13]. 

In  the  second  study,  performed  on  8  patients  with  PNS 
injury  and  mononeuropathy  with  tactile  allodynia,  3  ot  8 
patients  unexpectedly  had  belter  TI  perception  on  the 
affected  side  compared  to  their  unaffected  side  [4], 

Discussion.  Lesions  within  the  PNS  and  CNS  that  cause 
pain  are  often  associated  with  lemniscal  pathway  injury  as 
well.  For  example,  a  mononcucopathy  caused  by  a 
traumatic  injury  to  the  ulnar  nerve  will  often  injure  motor 
and  sensory  fibers.  The  injured  sensory  fibers  arc  destined 
for  both  the  pain-  and  temperature-mediating  reticular, 
spinothalamic  pathways  as  well  as  touch-,  pressure-,  and 
vibration-mediating  lemniscal,  posterior  column  pathways 
in  the  spinal  cord.  Lesions  in  the  CNS,  in  particular  those 
above  the  spinal  cord  level,  are  associated  with  both 
reticular  and  lemniscal  pathway  injury  due  to  the  close 
proximity  of  the  pathways  in  brainstem,  lateral  thalamic, 
and  suprathalamic  regions.  Spinal  cord  lemniscal 
pathways  arc  located  in  the  posterior  spinal  cord  whereas 
reticular  pathways  are  located  primarily  in  the  anterior 
spinal  cord.  None-lhe-lcss  spinal  cord  injuries  from 
conditions  such  as  infarcts,  trauma,  multiple  sclerosis 
plaques  and  others  will  often  times  produce  injury  to  both 
sensory  pathways. 

Even  though  spinal  cord,  brainstem  and  suprathalamic 
lesions  can  cause  central  pain  without  thalamic  injury, 
central  pain  is  dependent  upon  thalamic  modulation  [5]. 
The  accentuated  perception  of  the  TI  in  some  PNS  and 
CNS  neuropathic,  pain  patients  are  consistent  with  a 
proposed  central  disinhibition  theory.  This  theory  posits 
that  "tonic  activation  from  net  descending  facilitation  of 
supraspinal  sites"  |4],  results  in  improved  cortical 
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Fingertip  Tactile  Illusion-Producing  Device 


Display  Screen 


communications  Microresonators 

device 


A. 


B.  Microsolenoids 

Figure  2.  Microresonators  in  A.  fit  below  the 
surface  of  the  finger  indentation  of  the  frame.  The 
resonators  may  be  clusters  of  microelectrical 
mechanical  systems  (MEMS)  or  hexagonally 
stacked,  splayed  array  microsolenoids  as  shown 
in  B. 

processing  of  the  TI  [5,  6].  In  the  thalamic  injured  central 
pain  patients  with  TI  perseveration,  the  possible 
mechanism  of  this  phenomenon  may  include  TI 
aftersensations  released  from  the  disinhibited  cortex  or 
thalamus,  similar  to  the  tactile  movement  aftereffect 
mechanism  proposed  by  Hollins  and  Favorov  [7]. 

The  device  in  Figure  1  produced  cutaneous  stimulation 
over  several  dermatomes.  Theoretically  a  device  that 
would  stimulate  a  single  dermatome  will  provide  more 


precise  determination  of  peripheral  inputs  needed  to 
produce  this  illusion,  which  is  ultimately  caused  by  brain 
cortical  activity.  Newer  devices  in  development  will 
activate  a  smaller  area  of  skin  on  the  “tactile  rosette”  of 
the  index  finger,  in  a  single  dermatome,  and  will  spatially 
reduce  the  area  of  CNS  somatotopic  activation  (see 
Figures  2A  and  2B). 

The  index  fingertip  is  also  known  to  have  a  higher 
density  of  Pacinian  corpuscles  than  more  proximal 
cutaneous  sites  and  has  a  greater  primary,  and  possibly 
secondary,  parietal  cortical  representation  than  more 
proximal  cutaneous  dermatomes.  Localization  using  a 
single  dermatome  TI  device  on  the  fingertip  will  assist  in 
the  interpretation  of  CNS  imaging  using  functional 
magnetic  resonance  imaging  (/-MRI).  Therefore,  this  type 
of  device  will  be  helpful  in  determining  factors  that  may 
modulate  TI  discrimination  as  well  as  injury  associated 
sensitivity  changes  of  the  lemniscal  system.  This  may  also 
help  define  whether  the  newly  described  observation  of  TI 
perseveration  in  subsets  of  patients  with  central  pain  is 
dependent  on  multiple  peripheral  nerve  or  dermatomal 
activation,  or  spatial  summation  in  a  single  nerve  or 
dermatomal  distribution. 
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Abstract 

The  aim  of  this  paper  is  to  discuss  the  possibilities  of 
Perceptual  Filling-In  to  be  useful  in  haptic  displays, 
especially  for  perception  of  3D  form.  Examples  of  the 
functioning  of  this  mechanism  in  vision  and  natural 
haptics  are  reported,  and  obstacles  for  its  application  in 
haptic  displays  are  pointed  out,  such  as  the  lack  of 
extended  surface  contact,  restriction  (usually)  to  one 
point  of  contact  and  the  impossibility  of  performing 
exploratory  procedures  known  to  be  effective  in  natural 
haptics.  It  is  stated  that  the  haptic  information  obtain¬ 
able  in  the  haptic  displays  developed  so  far  is  not  suffi¬ 
cient  and  accurate  enough,  and  that  conditions  favour¬ 
ing  mechanism  such  as  Perceptual  Filling-In  should  be 
seriously  looked  for. 

1.  Introduction 

Visual  displays  can  mirror  the  real  world  to  a  very 
high  degree.  So  far,  haptic  displays  lag  far  behind  visual 
displays  in  efficiency  of  displaying  representations  of 
the  real  world.  The  success  of  visual  displays  depends 
both  on  the  ability  of  vision  to  handle  large  amounts  of 
complex  information  and  the  technical  possibilities  to 
imitate  this  information.  However,  the  visual  imitation  is 
not  perfect  but  utilises  also  the  fact  that  vision  can 
successfully  handle  also  optical  information  deviating 
from  the  information  presented  in  the  real  world.  This 
simplifies  the  technical  task  considerably.  One  example 
of  this  is  that  a  series  of  still  pictures  separated  by  black 
intervals  (film)  is  perceived  as  a  continuous,  constantly 
changing  event.  This  is  based  on  the  so-called  phi- 
phenomenon  (motion  perceived  from  still  pictures  under 
certain  conditions),  and  the  perception  is  further 
improved  by  the  perceptual  fusion  of  high-frequency 
flicker.  Another  example  is  TV  that  is  possible  because 
of  the  fact  that  the  eye  has  not  sufficient  spatio-temporal 
resolution  to  be  able  to  perceive  the  rapidly  moving  dots 
but  integrates  instead  the  scene  perceptually. 

The  aim  of  this  paper  is  to  discuss  the  possibilities  of 
utilising  related  options  for  haptic  displays,  especially 
for  3D  form  perception.  One  possibility  might  be  to 


utilise  a  perceptual  mechanism  that  is  important  for 
vision  and  natural  haptics  and  potentially  useful  also  for 
display  haptics.  The  mechanism  is  called  Perceptual 
Filling-In  (PFI).  The  emphasis  here  will  be  on  the 
problems  of  3D  form  perception  in  haptic  displays  and 
the  possible  contribution  of  PFI.  For  comparison,  some 
related  visual  phenomena  will  also  be  shortly  presented. 

2.  PFI  in  vision 

PFI  handles  situations  where  stimulation  is  incom¬ 
plete  or  totally  missing.  Well-known  examples  of  this 
mechanism  in  vision  are  the  PFI  of  stimulation  corre¬ 
sponding  to  the  blind  spot  in  the  eye,  as  well  as  some 
kinds  of  scotoma;  where  the  non-existing  visual  infor¬ 
mation  on  the  retina  corresponds  perceptually  to  a 
surface  of  the  same  kind  as  the  surrounding  areas.  Other 
examples  are  cases  where  whole  figures  and  objects  are 
perceived  from  incomplete  artefacts  when  gaps  in  the 
stimulation  are  filled  in  perceptually.  The  Gestalt 
psychologists  suggested  a  law  of  closure  to  account  for 
perception  in  such  cases.  Subjective  contours  (also 
called  illusory  or  cognitive  contours)  are  phenomena  of 
a  related  kind. 

2.1.  Visual  point  stimuli 

Especially  analogous  to  the  stimuli  presented  by 
present-day  haptic  displays  are  visual  point  stimuli.  It  is 
well  established  since  Johansson’s  motion  perception 
experiments  [11,  12]  that  two  points  in  motion  are 
perceived  under  many  conditions  as  the  endpoints  of  a 
rod.  These  results  have  been  further  elaborated 
concerning  patterns  consisting  of  a  few  dots  in  motion 
[2,  3].  Biological  motion  patterns  [13]  consisting  of  12 
moving  dots  are  perceived  as  human  beings  with  straight 
invisible  lines  of  constant  length  connecting  the  dots.  A 
point  line  pattern  can  also  be  perceptually  connected  by 
curved  lines,  for  instance  as  a  bending  line  [6,  19],  and  it 
has  been  demonstrated  that  the  number  of  points  in  such 
a  line  can  be  considerably  reduced  without  destroying 
the  perception  of  an  object  bending  smoothly  [14]. 
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3.  Compensation  with  Visual  Information 

The  problems  with  3D  form  perception  for  haptic 
displays  may  not  be  identified  as  problems  as  long  as 
haptic  and  visual  information  are  available  together. 
Vision  provides  an  overview  and  thereby  "fills  in"  the 
gaps  in  haptic  information.  It  also  suggests  interesting 
parts  of  the  scene  to  be  explored,  which  makes  haptic 
exploration  more  effective.  This  means  that  haptic  PFI  is 
less  needed  in  this  case. 

The  dominant  role  of  vision  for  the  judgement  of  3D 
form  of  real  objects  with  both  visual  and  haptic  infor¬ 
mation  available  has  been  experimentally  demonstrated 
[16].  When  vision  was  available  together  with  optional 
haptics,  the  observers  typically  used  only  vision  for 
judgements  of  geometric  properties,  such  as  size  and 
form.  Haptics  was  used  mainly  for  material  properties, 
such  as  texture  and  hardness.  A  similar  result  was 
obtained  in  a  study  concerning  virtual  objects  [1], 

4.  Perception  of  Form  by  Haptics  Alone 

During  natural  conditions  also  haptics  alone  has  a 
high  capacity.  In  his  studies  of  haptic  perception  of  real 
objects,  Gibson  [4,  5,  pp.  124-126)  found  that  the 
observers  can  be  quite  successful  under  these  conditions. 

When  observers  have  to  rely  on  haptics  alone  when 
using  haptic  displays,  as,  for  instance,  blind  people  have, 
the  situation  is  quite  different.  When  natural  and  virtual 
perception  of  3D  form  of  small  objects  of  simple 
geometric  forms  were  compared,  the  virtual  cases 
(explored  with  a  PHANToM)  were  found  to  be  both  less 
accurate  and  requiring  longer  exploration  times  [7]. 
Even  if  performance  was  improved  when  objects  were 
larger  (up  to  100  mm)  [7],  and  when  practice  was 
allowed  [10J,  the  performance  never  reached  the  level 
obtained  when  real  objects  were  involved.  Effective 
presentation  of  virtual  objects  with  a  complex  form  is 
still  a  difficult  problem  [8]. 

4.1.  The  Importance  of  Exploration 

The  hand  as  a  haptic  sense  organ  may  be  described  as 
a  touch  unit  consisting  of  five  fingers  as  suggested 
already  by  Katz  [15,  p.  59].  Considered  in  this  way,  the 
stimulus  properties  available  when  grasping  an  object  is 
at  each  moment  obtained  from  up  to  five  small  surfaces 
spatially  separated  by  (sometimes)  large  gaps  and  from 
the  kinaesthetic  information  from  muscles,  tendons  and 
joints  during  the  grasp.  The  haptic  perception  is  usually 
further  strengthened  by  exploratory  movements,  and 
total  perception  is  based  on  integration  over  time. 
According  to  Gibson  [4,  5],  observers  when  exploring 
tactually  curved  their  fingers  around  the  face  of  the 
object,  used  all  their  fingers  and  fitted  them  into  the 


cavities  and  used  oppositions  of  thumb  and  the  other 
fingers.  In  haptic  perception  of  real  objects  an  essential 
role  was  also  played  by  finger  directions  and  finger 
spans,  the  "bone  space"[5,  p.  125]. 

Lederman  &  Klatzky  [17]  presented  more  detailed 
studies  of  the  exploratory  procedures  in  haptics. 
Concerning  perception  of  form/shape,  they  identified 
two  main  exploratory  procedures:  enclosure,  which  is 
performed  by  more  than  one  finger  and  often  both 
hands,  and  contour  following,  which  may  be  executed  by 
one  or  more  fingers.  These  exploratory  procedures  may 
be  performed  in  sequence.  The  efficiency  of  these  proce¬ 
dures,  also  for  quite  complex  object  forms,  is  remarkable 
concerning  both  accuracy  and  speed. 

5.  PFI  in  Haptic  Displays 

The  importance  of  PFI  for  visual  perception  has  been 
demonstrated.  Stimulation  from  two  or  more  points  may 
give  perception  of  an  object  filling  the  space  between 
these  points.  The  present  discussion  concerns  the  possi¬ 
bilities  of  a  similar  mechanism  in  haptics,  especially  for 
the  perception  of  3D  form  of  virtual  objects.  That  such 
forms  can  be  perceived  is  a  critical  feature  for  the 
usefulness  of  haptic  displays  when  haptics  is  the  only 
form  of  information  available.  If  form  perception  under 
this  condition  is  not  sufficiently  accurate  and  fast,  the 
use  of  such  a  display  without  visual  assistance  may  be 
questioned.  It  is  therefore  important  to  try  all  potential 
options  to  increase  accuracy  and  speed  in  these  cases. 

5.1.  2D  Form 

That  PFI  can  work  also  in  virtual  haptics  is  demon¬ 
strated  by  the  relative  success  of  displays  consisting  of  a 
matrix  of  tactile  point  stimuli,  such  as  the  Optacon.  The 
raised  points  of  this  kind  of  display  are  typically 
perceived  as  lines  and  areas  in  2D.  It  should  be 
observed,  however,  that  in  these  cases  a  larger  skin 
surface  is  in  contact  with  the  display,  and  some  displays 
allow  also  the  use  of  more  than  one  finger. 

5.2.  3D  Form 

The  perception  of  3D  form  by  haptics  is  more 
complicated  than,  for  instance,  the  perception  of  2D 
form  and  of  texture.  A  uniform  texture  (the  microstruc¬ 
ture  or  high  frequency  properties)  can  be  explored  rather 
similarly  in  natural  and  display  haptics,  by  lateral 
movements  over  the  surface,  and  the  resulting  perceived 
properties  of  the  surface  may  be  quite  similar  in  the  two 
cases  [9].  In  contrast,  the  exploratory  procedures  avail¬ 
able  when  the  task  is  to  judge  global  form  (the  macro¬ 
structure  or  low  spatial  frequency  properties)  are  quite 
different  in  natural  and  display  contexts. 
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5.3.  Differences  in  Exploratory  procedures 

between  Natural  and  Display  Haptics 

In  natural  haptics  the  exploratory  procedures  pick  up 
information  from  both  extended  skin  surfaces  and  from 
the  kinaesthetic  sensory  system.  When  a  real  object  is 
explored  for  judgment  of  form,  the  result  is  that  an 
object  is  perceived  that  fills  the  space  between  the 
contact  points.  The  goal  for  haptic  displays  should  be  to 
construct  them  such  that  sufficient  information  can  be 
picked  up  to  make  perception  of  a  virtual  object  between 
the  contact  points  possible. 

Important  features  of  most  haptic  displays  are  that  the 
contact  between  observer  and  virtual  object  is  restricted 
to  a  minimal  point  of  a  virtual  object  and  is  mainly  based 
on  kinaesthetic  information.  3D  form  may  be  perceived 
in  this  situation,  but  not  with  the  same  accuracy  and 
speed  as  in  real  haptics  [7].  That  the  elimination  of 
spatially  distributed  information  at  the  skin  is  an 
important  factor  was  demonstrated  by  experiments 
where  a  similar  restriction  was  applied  to  real  object 
exploration  [18].  The  results  in  these  experiments  were 
that  several  haptic  functions  were  substantially  impaired. 

Information  about  3D  form  of  a  virtual  object  can  in 
these  cases  be  obtained  only  over  time.  Such  explora¬ 
tion,  for  instance  with  a  one-point  stylus,  is  not  common 
in  natural  haptics.  Exploring  with  a  thimble  is  more 
similar  to  natural  exploration,  but  in  natural  haptics  it  is 
not  the  only  available  exploratory  procedure. 

5.4.  Can  Multi-Finger  Displays  Solve  the 
Problems? 

For  PFI  to  function  in  simultaneous  stimulation  there 
must  be  at  least  two  points  of  contact.  For  perception  of 
3D  form  there  ought  to  be  at  least  three.  As  with  visual 
point  stimuli  it  can  be  expected  that  stimulation  chang¬ 
ing  over  time  increases  the  efficiency.  With  a  one-point 
display  it  is  not  possible  to  enclose  the  virtual  object  in 
order  to  get  information  about  its  form.  An  important 
exploratory  procedure  is  thus  not  available. 

Haptic  exploration  of  a  real  object  is  an  effective  but 
complex  activity,  which  may  be  difficult  to  mirror  for 
virtual  objects.  The  object  is  moved  around  by  fingers  of 
one  hand  or  both  hands,  which  provides  a  richness  of 
both  cutaneous  and  kinaesthetic  information  far  from  the 
poverty  of  information  obtained  when  only  one  small 
point  at  a  time  is  in  contact  with  the  object. 

There  are  efforts  to  provide  the  observer  with  more 
contact  points.  Using  two  PHANToMs,  for  instance,  is 
possible,  and  a  six-degrees-of-freedom  PHANToM  has 
been  constructed  (http://www.sensable.com'),  as  well  as 
multi-finger  displays,  for  instance  the  CyberGIove  and 
the  CyberForce  (http://www.virtex.com').  Such  devices 
increase  the  number  of  contact  points,  but  to  what  extent 


they  improve  the  perceptual  potentials  of  virtual  3D 
form  perception  does  not  seem  to  have  been  studied.  A 
six-degrees-of-freedom  stylus  gives  an  opportunity  to 
get  more  than  one  contact  point.  However,  to  what 
extent  is  information  from  minimal  points  along  a  stylus 
useful  forjudging  the  form  of  a  virtual  object? 

It  remains  also  to  be  studied  if  multi-finger  displays 
offer  stimulation  sufficiently  similar  to  that  obtained  in 
natural  haptics  to  make  effective  3D  form  perception 
possible.  The  availability  of  information  from  several 
minimal  points  may  not  be  sufficient.  The  exact  form  of 
the  kinaesthetic  information  is  probably  also  critical.  It  is 
still  an  open  question  if  such  information  is  provided  via 
the  present  multi-finger  displays  with  force  feedback  in 
a  few  directions. 

6.  Conclusion 

Natural  haptics  alone  is  quite  accurate  and  fast  in 
perceiving  3D  form,  but  haptic  displays  developed  so  far 
are  not  as  successful.  They  do  not  mirror  very  closely 
the  haptic  information  obtained  naturally,  which  means 
that  the  information  they  provide  is  not  sufficient  and 
adequate  enough.  The  finding  of  conditions  favouring 
haptic  PFI  ought  to  be  seriously  considered.  It  works  in 
vision  and  natural  haptics.  If  such  conditions  can  be 
found,  the  performance  of  users  may  be  improved  and 
the  technical  development  of  new  devices  simplified.1 
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Abstract 

Studies  in  motor  psychophysics  have 
documented  that  the  tangential  velocity  of 
movement  decreases  with  increasing  curvature 
of  the  path  being  followed.  Views  on  the  origin 
of  this  relation  range  from  higher  order  motion 
planning  processes  such  as  trajectory  selection 
according  to  a  principle  of  minimisation  of 
mean  squared  rate  of  change  of  acceleration 
(jerk)  to  lower  level  motor  constraints,  for 
instance  sinusoidal  constraint  on  joint  motion. 
We  have  examined  finger-scanning  movements 
in  which  eight  subjects  traced  repeatedly 
round  various  raised  outline  shapes.  We  were 
interested  to  see  whether  velocity  would  be 
related  to  curvature  from  the  outset, 
suggesting  low  level  sensorimotor  constraints, 
or  whether  the  relation  would  emerge  with 
experience,  as  a  result  of  increased  trajectory > 
planning.  We  found  evidence  of  a  relation 
between  speed  and  curvature  from  the  outset, 
in  support  of  lower  level,  rather  than 
trajectory  planning,  constraints. 

1 .  Introduction. 

A  fundamental  law  in  motor  psy  chophysics 
is  the  one  third  power  relation  between 
movement  speed  (or  more  precisely,  tangential 
velocity)  and  radius  of  curvature  of  the  path 
followed: 

speed  = 

radius_of_curvature*exp(l/3). 

The  smaller  the  radius  of  curvature  (the  greater 
the  curvature)  the  lower  the  speed. 

Although  a  relation  between  speed  and 
curvature  has  been  acknowledged  for  over  a 
century'  (Jack,  1895)  there  has  been 
considerable  growth  in  interest  since 
Lacquaniti  et  al  (1983)  provided  a  thorough 
quantification  of  the  law  in  drawing 
movements.  The  law  holds  for  writing  as  well 
as  drawing  movements.  It  holds  for 
movements  in  the  plane  or  in  3d  (Schall  and 
Stemad,  2001)  and  for  drawing  motions 
directed  by  3-d  isometric  force  control 
(Massey  et  al,  1992).  The  one-third  power  law 
has  been  demonstrated  in  children  as  well 
adults  although  the  exponent  changes  with  age 


(Viviani  and  Schneider,  1991).  It  has  been 
found  for  mechanically  constrained  or  free 
form  movements,  with  or  without  visible  trace 
of  the  path  traced  out  (Viviani,  BaudBovy  & 
Redolfi,  1997)  and  for  eye  movements  (de 
Sperati  and  Viviani,  1997).  When  participants 
were  guided  around  an  elliptical  track  by  a 
robot,  which  followed  a  different  relation 
between  speed  and  curvature,  subsequent  free 
reproduction  of  the  track  without  any  guide 
conformed  to  the  one  third  power  law  (Viviani 
et  al  1997). 

The  one-third  power  law  has  been  attributed 
to  motion  planning  processes  with  trajectory 
selected  according  to  a  principle  of 
minimisation  of  jerk,  the  third  derivative  of 
position,  in  order  to  obtain  maximally  smooth 
movements  (Flash  and  Hogan,  1985;  see  also 
Wann,  Nimmo-Smith  &  Wing,  1988;  Viviani 
and  Flash  1995).  An  alternative  interpretation 
is  that  the  relation  reflects  a  fundamental 
constraint  on  motor  control.  For  example,  in 
handwriting  it  has  been  attributed  to  the 
combination  of  oscillatory  tendencies  related 
to  x-  and  y-directions  which  may  reflect  joint 
rather  than  hand  space  constraints  (Schall  and 
Sternad,  2001). 

In  the  above  studies  the  movements 
analysed  were  made  to  spatial  targets  which 
the  participant  had  opportunity  to  plan.  We  ask 
whether  the  one-third  power  law  applies  to 
tactile  exploration  where  a  continuous  curve 
target  path  (circle  and  ellipses)  has  to  be 
discovered.  When  finger  scanning  movements 
are  used  to  trace  an  outline  shape  without 
visual  guidance,  it  is  reasonable  to  expect  that 
the  demands  of  tactile  and  proprioceptive 
information  pick-up  will  result  in  relatively 
slow  movements.  But,  will  the  slowing  be 
related  to  curvature?  If  the  one-third  power  law 
arises  from  planning  processes,  the  relation 
between  speed  and  curvature  would  only  be 
expected  to  appear  as  information  about  the 
path  becomes  available  from  the  haptic  input. 
However,  if  the  relation  arises  from  low  level 
sensorimotor  constraints,  these  would  be 
expected  to  scale  down  to  lower  speeds  used  in 
first  exploring  the  shape.  The  results  that  we 
report  in  this  paper  show  differences  in  speed 
at  the  major  and  minor  axes  of  the  ellipses 
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consistent  with  the  power  law  thus  supporting 
sensorimotor  rather  than  motion  planning 
constraints. 

2.  Method. 

2.1  Subjects. 

Eight  subjects  (4  male,  4  female,  age  range 
19  to  22  years,  right-handed  for  writing), 
students  of  The  University  of  Birmingham, 
participated  on  a  voluntary  basis. 

2.2  Apparatus. 

Motion  of  a  reflective  marker  placed  on  the 
nail  bed  of  the  index  finger  of  the  right  hand 
was  tracked  at  100  Hz  with  a  3 -camera 
infrared  motion  tracking  system  (Qualisys 
ProReflcx).  The  system  calibration  yielded  0.2 
mm  resolution  over  a  working  area  of  40  x  40 
cm.  Outline  shapes  were  constructed  from  (4 
mm  diameter)  plastic  coated  electrical  cable 
glued  to  a  stiff  backing  board.  They  comprised 
a  circle  (radius  12  cm),  a  wide  ellipse  (angular 
eccentricity  45°  and  linear  eccentricity  24  cm) 
and  a  narrow  ellipse  (25°  and  24  cm). 

2.3  Task  and  procedure. 


2.5  Design. 

A  repeated  measures  design  was  used  with 
fixed  order  of  outline  shapes  but  with  the  order 
varied  across  subjects  so  that  each  shape 
appeared  in  first  position  approximately  the 
same  number  of  times. 

3.  Results. 

Illustrative  time  and  spatial  finger  tip 
trajectories  for  one  subject  tracing  the  wide 
ellipse  oriented  in  the  coronal  plane  without 
(above)  and  with  (below)  vision  are  shown  in 
Figure  1 .  The  time  waveforms  show  the 
greater  speed  achieved  in  the  presence  of 
vision.  The  increased  speed  is  associated  with 
the  development  of  a  series  oflocal  speed 
maxima  and  these  can  be  seen  to  be  aligned 
with  extremes  in  y-displacement  associated 
with  low  curvature  at  each  end  of  the  minor 
axis  of  the  ellipse.  Emergence  of  similar 
maxima  may  be  seen  in  the  absence  of  vision, 
although,  in  this  case,  only  after  a  couple  of 
passes  around  the  ellipse.  In  this  example,  the 
speed  maxima  are  much  more  poorly  defined 
than  in  the  vision  condition  although  there  is  a 
suggestion  they  align  with  the  y  extremes. 


The  subject  was  seated  at  a  table  with  the 
outline  shape  placed  on  it  so  that  the  semiaxes 
(of  the  ellipse)  were  aligned  to  the  subject’s 
coronal  (left-right)  and  sagittal  (anterior- 
posterior)  planes  and  corresponded  to  the  x  and 
y  axes  of  the  motion  tracking  system. 
Instructions  were  given  to  use  the  right  index 
finger  to  trace  each  shape  at  a  comfortable 
speed  for  30  s  in  an  anti-clockwise  direction 
with  the  pad  of  the  index  finger  on  top  of  the 
wire.  The  subject  was  instructed  not  to  rest  the 
finger  tip  on  the  surface,  nor  to  use  the  wire  as 
a  guide  “fence”,  nor  was  any  other  part  of  the 
hand  or  arm  allowed  to  touch  the  shape  or 
surface  of  the  table.  Three  successive  trials 
were  run  with  each  shape  before  proceeding  to 
the  next  shape.  The  ellipses  were  presented, 
first  with  long,  then  w  ith  short,  axis  parallel  to 
the  coronal  plane.  All  trials  were  first  run  with 
the  subject’s  eyes  closed  and  subsequently 
repeated,  one  trial  per  condition,  with  the 
subject  allowed  full  vision. 

2.4  Data  processing. 

Tangential  velocity  (speed)  at  x-  and  y- 
extremes  was  determined  from  the  finger  tip 
trajectory'  after  low-pass  filtering  (Butterworth 
2-order  10  Hz)  the  x  and  y  coordinate  streams. 


100  mm 


5  s 


100  mm 


Figure  I:  Time  (left)  and  spatial  (right) 
trajectories  for  two  single  trials  in  which  the 
subject  traced  the  wide  ellipse  by  touch  alone 
(above)  or  with  vision  (below).  Displacements 
(x  dashed,  y  thin  continuous  lines),  speed 
(thick  line)  are  plotted  on  the  same  vertical 
scale  but  note  the  difference  in  the  time  scale 
reflecting  the  doubling  in  speed  with  vision. 

The  number  of  cycles  completed  in  each  30 
s  trial  shown  in  Figure  2  provides  an  indication 
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of  the  average  speed  of  finger  scanning,  It  can 
be  seen  that,  for  each  outline  shape  and  ellipse 
orientation,  the  speed  progressively  increases 
across  the  3  trials  without  vision,  although 
remaining  slower  than  with  vision.  This  w'as 
confirmed  by  reliable  main  effects  of  trial 
(F3,21=53.49,  p<.01)  and  stimulus 
(F4, 28=5. 65,  p<01)  in  repeated  measures 
ANOVA.  The  interaction  between  trial  and 
stimulus  was  also  reliable  (F12, 84=4.92, 

p<01). 


Figure  2.  Average  (N=8  subjects)  of  the 
number  of  cycles  completed  in  each  30  s  trial 
as  a  function  of  trial  (trials  1-3  eyes  closed,  4 
eyes  open).  Five  stimulus  types  are  represnted 
in  the  five  sets  of  lines  -  circle  (coded  111), 
wide  (12.)  or  narrow  (13.)  ellipse  with  the 
ellipses  oriented  with  major  axis  parallel  to  the 
coronal  (.. I )  or  saggital  (.2) plane. 


rxl  ryt  rx2  ry2  rx3  ry3 


loop  no  -  (xfy) 


Figure  3.  Speed  as  a  function  of  3  successive 
loops  (measures  taken  at  x  andy  extremes  in 
alternation)  around  the  circle.  The  four  curves 
relate  to  no  vision  trials  1-3  (lower  speeds) 
and  vision  trial  4  (highest  speed). 

The  measured  speed  at  x  and  y 
displacement  extremes  in  tracing  around  the 
circle  for  the  first  three  complete  loops  is 


shown  in  Figure  3.  An  increase  in  speed  over 
haptic  trials  is  clear  although  the  trials  with 
vision  remain  faster.  There  is  also  an  increase 
in  speed  evident  over  the  3  successive  loops 
around  the  circle. 

The  speed  at  major  and  minor  axes  of  the 
wide  and  narrow  ellipses  is  shown  in  Figure  4. 
An  increase  in  speed  over  haptic  no  vision 
trials  is  clear  although  the  trials  with  vision 
remain  faster.  There  is  also  a  slight  increase  in 
speed  evident  over  the  3  successive  loops.  It 
can  also  be  seen  that  the  speed  is  greater  for 
the  lower  curvature  minor  axis  of  the  ellipse 
which,  depending  on  whether  the  ellipse  is 
oriented  with  coronal  (LR)  or  saggital  (AP) 
planes,  corresponds  to  the  y  (in  LR)  or  x  (in 
AP)  displacement  extremes. 

wid«*lflpf*LR  wld*  AP 

4C0 


Figure  4.  Speed  as  a  function  of  3  successive 
loops  (measures  taken  at  x  andy  extremes  in 
alternation)  around  the  ellipse.  In  each  of  the 
four  plots  the  curves  relate  to  no  vision  trials 
1-3  (lower  speeds)  and  vision  trial  4  (highest 
speed).  The  separate  plots  relate  to  ellipse 
shape  (top  row  -  wide;  bottom  row  -  narrow) 
and  orientation:  (left  column  coronal  LR;  right 
column  saggital  (AP). 

Separate  repeated  measures  ANOVAs  for 
each  stimulus  generally  revealed  significant 
main  effects  (9  out  of  12  cases)  of  trials,  loops 
and  axis  (F3,21,  F2,2 1  and  Fl,7,  ranging  8.57- 
38.88, p<  01,  3.89-16.86, p<.05  and 32.80- 
135.03,  p<  01  respectively).  Interactions  terms 
involving  loop  number  x  axis  and  trial  x  axis 
(F2,I4  and F3,21  ranges 6.34-32.51,  p<01 
and  5.71-17.82,  p<.01  respectively)  were 
significant  in  7  out  of  8  cases  due  to  there 
being  less  difference  between  speed  at  high 
and  low  curvatures  at  x,y  extremes  in  the 
initial  trial  and  initial  loop.  However,  none  of 
the  three-way  trial  x  loop  x  axis  interactions 
was  significant. 
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4.  Conclusion. 

Numerous  studies  have  shown  a  power  law 
relation  exists  between  curvature  and 
tangential  velocity  (speed).  For  the  first  time 
we  have  examined  haptic  exploratoiy 
movements  in  which  subjects  used  the  index 
finger  to  trace  out  a  raised  outline.  We 
considered  two  alternative  predictions.  Under 
the  motion  planning  hypothesis  we  predicted 
that  in  the  early  stages  speed  would  be 
constant.  Only  with  knowledge  of  shape 
gained  from  haptic  exploration  would  the 
relation  between  speed  and  curvature  emerge. 
Under  motor  constraints  we  expected  that  the 
relation  between  speed  and  curvature  would 
hold  from  the  outset,  even  at  the  lower  speeds 
of  initial  exploration. 

The  results  showed  large  increases  in  the 
speed  of  scanning  with  the  index  finger  tip 
with  growing  experience  of  moving  around 
circular  and  elliptical  outline  shapes.  However, 
a  relation  between  speed  and  curvature  was 
observed  from  the  very  beginning,  although 
the  difference  in  speed  at  high  and  low 
curvatures  was  somewhat  less  in  initial  loop 
around  the  ellipses  and  on  the  initial  trial.  We 
therefore  suggest  that  the  one  third  power  law 
reflects  low  level  sensorimotor  processes 
involved  in  movement  execution  and  not 
higher  level  motion  path  planning  processes. 
However,  this  conclusion  must  be  tempered  by 
the  following  observations: 

1)  in  this  preliminary  analysis  we  have  not 
documented  the  continuous  relation  between 
speed  and  curvature,  but  simply  focussed  on 
curvature  extremes. 

2)  in  this  study,  the  regularity  of  the  geometric 
shapes  (circles  and  ellipses)  may  have  led  to 
rapid  recognition  of  the  particular  form  being 
traced.  This  means  that  subjects  may  have 
been  able  to  form  a  mental  image  of  the 
general  shape  class,  sufficient  to  support 
motion  planning  very  soon  after  contact.  In 
further  experiments  we  are  evaluating  haptic 
scanning  of  irregular  raised  outline  paths  of 
non-closed  fonn. 
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Abstract 

Haptic  tasks  call  upon  diverse  processing  resources 
within  the  human  cognitive  architecture.  In  order  to 
develop  appropriate  interfaces  it  may,  in  consequence,  be 
advantageous  to  adopt  a  holistic,  psychologically  based 
approach  to  its  study.  A  candidate  theory  through  which 
this  might  be  achieved  is  Barnard’s  Interacting  Cognitive 
Subsystems  (ICS).  ICS  is  a  comprehensive  theory  of 
cognition  and  affect.  It  abstracts  across  the  details  of 
individual  types  of  processing  in  order  to  permit  a  generic 
understanding  of  the  systemic  principles  operating  across 
the  human  cognitive  architecture. 

ICS  is  briefly  described,  followed  by  an  example 
scenario,  in  order  to  illustrate  how  it  might  be  applied  to 
the  understanding  of  haptic  tasks.  In  addition,  the 
structure  of  mental  representations  within  individual 
types  of  processing  is  outlined.  The  importance  of  system- 
wide  cognitive  processes  is  considered  before  finally,  an 
outline  is  given  of  the  aims  and  objectives  of  the  author’s 
future  work. 


1.  Introduction 

Effective  haptic  interaction  frequently  requires 
complex  cognitive  activity  on  behalf  of  the  user.  For 
example,  touch-based  perceptions  may  be  affected  by 
various  factors.  These  include  stored  high-level 
knowledge  [1],  the  particular  object  properties  sensed  [2], 
availability  of  cues  for  object  identification  [3],  perceptual 
bias  by  other  modalities  [4,  5,  6]  and  cross-modal 
attention  [7,  8].  The  sense  of  touch  is,  therefore, 
dependent  upon  diverse  cognitive  resources  and,  as  a 
result,  it  may  be  useful  to  adopt  a  holistic,  psychologically 
based  approach  to  its  study. 

The  present  paper  outlines  initial  steps  towards  the 
achievement  of  a  holistic,  haptic  psychological  schema, 
based  upon  Barnard’s  Interacting  Cognitive  Subsystems 
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(ICS)  [9J.  ICS  is  a  comprehensive  theory  of  cognition  and 
affect.  It  adopts  a  systemic  approach  in  which  information 
flows  within  a  highly  parallel  and  modular  architecture  of 
distributed  cognitive  resources.  As  behaviour  arises  out  of 
the  coordinated  operation  of  its  constituent  parts,  a  major 
advantage  of  ICS  is  that  cognition  and  affect  can  be 
considered  within  an  overall  psychological  context, 
therefore  making  it  ideally  suited  to  inform  reasoning 
about  complex  haptic  interaction. 

A  brief  outline  of  ICS  will  now  be  provided  followed 
by  a  description  of  some  of  its  implications  for  haptic 
processing.  For  a  detailed  description  of  the  ICS 
architecture  interested  readers  should  refer  to  previous 
publications  in  which  it  has  been  outlined  fully  [10,  11, 
12, 13]. 

2.  ICS 

ICS  postulates  nine  subsystems,  each  of  which  differs 
in  terms  of  types  of  information  dealt  with,  level  of 
representational  abstraction  and  form  of  processing  code 
(Figure  3).  Although  individual  subsystems  are  associated 
with  different  types  of  subjective  experience,  they  each 
share  the  same  basic  internal  processing  architecture, 
including  a  local  memory  ( Image  Record)  and  a  number 
of  parallel  processes  through  which  information  is 
transformed  from  one  form  of  subsystem  code  to  another 
(Figure  1).  This  permits  subsystems  to  operate  together 
as  a  cohesive  system.  Behaviour  results,  therefore,  from 
the  flow  of  information  through  processing  configurations 
within  the  overall  cognitive  architecture. 

The  transformations  permissible  have  been  derived 
systematically  according  to  evidence  available  from 
experimental  psychology  and  subjective  plausibility 
(Figure  2).  ICS  has  been  applied  within  many  theoretical 
areas  including  the  understanding  of  depression  [10,  11], 
mental  number  generation  [14]  and  the  analysis  of 
cognitive  processing  in  complex  user  tasks  [15]. 
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3.  ICS  and  the  haptic  sense 

ICS  may  be  used  to  predict  various  mental  and 
behavioural  effects  for  users  engaging  in  haptic  tasks, 
with  many  implications  for  effective  interface  design. 
Most  importantly,  the  model  makes  it  possible  to  reason 
about  the  flow  of  information  through  the  human 
cognitive  architecture  between  sensation  of  a  stimulus  and 
production  of  a  response.  For  example,  if  a  user  is 
presented  with  an  apparently  simple  haptic  task  (such  as 
to  obtain  as  much  information  as  possible  about  an 
unknown  felt  object)  a  characteristic  flow  of  information 
processing  may  be  predicted. 


Image  Record 
of  the  Subsystem 
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j  Input  Array  | _ 

|  jlfipp:# 
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Figure  1 

The  internal  architecture  of  an 

individual  subsystem 

3.1  Sensation  and  premotor  action  plans 

Information  will  arrive  within  ICS  via  the  bodily 
receptor  organs  at  the  BS  ( Body  State )  subsystem. 
Representations  within  BS  are  based  upon  information 
about  the  type  of  stimulation  experienced  (e.g.  cutaneous 
pressure,  temperature,  muscle  tension  etc.),  its  location, 
intensity  and  so  on.  The  resultant  subjective  experience 
takes  the  form  of  bodily  sensations  including  pressure, 
pain  and  positions  of  parts  of  the  body.  Three  parallel 
transformations  from  BS  code  to  other  types  of  subsystem 
code  will  then  occur,  two  of  which  are  central  to  the 
understanding  of  haptic  experiences. 

A  transformation  from  BS  to  the  LIM  (Limb) 
subsystem  produces  representations  based  upon 
information  about  bodily  forces,  target  positions  and 
timing  of  skeletal  musculatures.  Subjectively  speaking  the 
experience  is  of  ‘mental’  physical  movement.  Through 
the  combined  action  of  the  BS  and  LIM  subsystems  it 
therefore  becomes  possible  for  a  comparison  to  be  made 
between  actual  bodily  states  and  planned  bodily  states. 
Once  executed  as  physical  actions  upon  the  world,  the 


results  can  then  be  assessed  through  subsequent  sensation, 
thereby  forming  a  loop  through  which  accurate  bodily 
adjustments  and  readjustments  may  be  achieved. 


Figure  2.  Schematic  representation  of  ICS 
including  the  full  range  of  permissible  trans¬ 
formation  processes  (grey  lines) 

3.2  Sensation  and  high-level  cognition 

A  parallel  transformation  from  BS  to  the  IMP 
(Implicational)  subsystem  results  in  schematic 
representations  of  sensory  experience.  For  example, 
abstract  ideational  properties,  such  as  surface  ‘hardness’ 
or  ‘smoothness’,  may  be  subjectively  experienced.  It  is, 
therefore,  possible  for  haptic  information  to  access  the 
level  of  meaning  in  a  relatively  direct  manner  within  ICS. 
The  result  may  be  an  ability  to  use  touch  to  categorise 
sensation  within  different  types  of  schematic  concepts.  In 
order  for  the  user  to  take  advantage  of  haptic  experiences 
in  other  ways,  however,  further  processing  is  necessary  in 
terms  of  transformations  to  other  subsystems. 

For  example,  a  transformation  from  IMP  to  the  PROP 
(Propositional)  subsystem  results  in  representations  based 
upon  semantic  fact,  the  subjective  experience  being  of 
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PERIPHERAL  SUBSYSTEMS 

a)  Sensory 

ill  Acoustic  (AC): 

Sound  frequency  (pitch),  timbre,  intensity  etc. 

Subjectively,  what  we  ‘hear  in  the  world’. 

12)  Visual  (VIS): 

Light  wavelength  (hue),  brightness  over  visual  space  etc. 
Subjectively,  what  we  ‘see  in  the  world’  as  patterns  of  shapes 
and  colours. 

(3)  Body  State  (BS): 

Type  of  stimulation  (e.g.,  cutaneous  pressure,  temperature, 
olfactory,  muscle  tension),  its  location,  Intensity  etc. 
Subjectively,  bodily  sensations  of  pressure,  pain,  positions  of 
parts  of  the  body,  as  well  as  tastes  and  smells  etc. 

b)  Effector 

(4)  Articulatory  (ART): 

Force,  target  positions  and  timing  of  articulatory 
musculatures  (e.g.,  place  of  articulation). 

Subjectively,  our  experience  of  subvocal  speech  output. 

(5)  Limb  (LIM): 

Force,  target  positions  and  timing  of  skeletal  musculatures. 
Subjectively,  'mental'  physical  movement. 


CENTRAL  SUBSYSTEMS 

c)  Structural 

(6)  Morphonolexical  (MPL): 

An  abstract  structural  description  of  entities  and  relationships 
in  sound  space.  Dominated  by  speech  forms,  where  it 
conveys  a  surface  structure  description  of  the  identity  of 
words,  their  status,  order  and  the  form  of  boundaries 
between  them. 

Subjectively,  what  we  ‘hear  in  the  head’,  our  mental  ‘voice’. 

(7)  Object  (OBJ): 

An  abstract  structural  description  of  entities  and  relationships 
in  visual  space,  conveying  the  attributes  and  identity  of 
structurally  integrated  visual  objects,  their  relative  positions 
and  dynamic  characteristics. 

Subjectively,  our  ‘visual  imagery.' 

d)  Meaning 

(8)  Propositional  (PROP): 

A  description  of  entities  and  relationships  in  semantic  space 
conveying  the  attributes  and  identities  of  underlying  referents 

and  the  nature  of  relationships  among  them. 

Subjectively,  specific  semantic  relationships  (‘knowing  that'). 

(9)  Implicational  (IMPLIC): 

An  abstract  description  of  human  existential  space, 
abstracted  over  both  sensory  and  propositional  input,  and 
conveying  ideational  and  affective  content:  schematic  models 
of  experience. 

Subjectively,  'senses’  of  knowing  (e.g.,  ‘familiarity’  or  'causal 
relatedness’  of  ideas),  or  of  affect  (e.g.,  apprehension, 
desire). 


Figure  3.  The  functions  of  the  nine  ICS 
subsystems  (adapted  from  Barnard  and  May, 
1995  [13]) 


meaningful  statements  about  the  sensory  experience  such 
as  This  is  a  hard,  smooth  surface’.  Two  further  parallel 
transformations  then  occur.  A  link  direct  from  PROP  back 
to  IMP  results  in  reciprocal  loops  of  processing  between 
the  two  subsystems.  As  a  result,  meaningful  thought 
becomes  possible.  Indeed,  the  transformation  processes 
between  PROP  and  IMP  form  the  processing  basis  for 
many  types  of  high-level  cognitive  tasks  and  their 
combined  action  is,  therefore,  termed  the  Central  Engine 
of  Cognition.  Furthermore,  representations  of  tasks 
themselves  will  be  processed  within  IMP  and  PROP, 
thereby  allowing  representations  (achieved  through 
transformations  from  the  senses)  to  be  related  to  the 
active  aims  and  goals  of  the  perceiver.  In  the  present 
instance  the  task  may  result  in  representations  at  PROP 
such  as  ‘learn  new  tilings  using  touch’  and,  at  IMP, 
categorizations  of  information  as  ‘old’  or  ‘new’.  The  two 
subsystems  will,  therefore,  complement  each  other  in 
decision  making  such  that  new  information  is  processed 
and  old  information  is  not. 

3.3  Visuo-spatial  and  auditory  Imagery 

In  parallel  with  the  initial  transformation  from  PROP 
to  IMP  a  second  process  also  occurs  to  the  OBJ  ( Object ) 
subsystem.  Here  visuo-spatial  representations  become 
active,  the  result  being  mental  imagery  based  upon  PROP 
output.  Whilst  this  may  permit  the  activation  of  a  mind’s 
eye  image,  of  the  felt  experience,  it  is  rendered  indirect  by 
the  need  to  pass  first  through  IMP  and  PROP  after  the 
initial  BS  representation.  Information  sensed  via  touch 
may,  therefore,  permit  the  achievement  of  effective  visuo- 
spatial  imagery  but  only  after  processing  at  the  level  of 
meaning. 

In  addition,  a  third  transformation  from  PROP  to  the 
MPL  ( Morphonolexical )  subsystem  produces 

representations  characterized  in  terms  of  auditory  imagery 
and,  in  particular,  that  relating  to  speech.  The  subjective 
experience  for  the  user  will  be  of  an  'inner  voice’  -  an 
internal  verbal  dialogue  based  upon  the  output  received 
from  PROP.  As  conduction  of  the  haptic  task  progresses 
this  will  serve  as  a  subvocal  commentary  upon  the  state  of 
the  interaction.  Via  a  transformation  back  to  PROP 
representations  processed  at  MPL  may,  in  turn,  influence 
the  level  of  meaning.  The  user  might,  therefore,  engage  in 
an  internal  discussion  with  him/herself  through  the 
reciprocal  action  of  MPL  and  PROP. 

3.4  Stored  information 

Haptic  processing  is  also  dependent  upon  stored 
representations  at  each  of  the  various  subsystems 
involved.  If  incoming  information  matches  or  activates 
representations  within  the  image  record  of  a  given 
subsystem,  the  data  held  within  memory  may  be  used  to 
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elaborate  that  derived  from  the  immediate  sensation,  the 
result  being  relatively  fast  and  detailed  representations.  If, 
however,  stored  representations  are  either  unavailable  or 
irrelevant,  processing  will  be  constrained  to  v/ork  upon 
the  sensory  input  alone.  In  such  circumstances  haptic 
information  will  require  increased  processing  time  due  to 
inherent  restrictions  on  the  degree  of  parallel  processing 
its  dedicated  sense  receptors  are  able  to  accomplish.  In 
consequence  it  may  be  necessary  to  take  advantage  of  a 
further  transformation  process  from  OBJ  to  LIM,  the 
result  of  which  being  an  influence  of  visuo-spatial 
imagery  upon  planned  bodily  movements.  Once 
physically  performed,  the  results  of  bodily  actions  can 
then  be  sensed,  providing  a  loop  between  cognition  and 
the  sensory  environment. 

3.5  High-level  cognition  and  premotor  plans 

A  parallel  transformation  process  from  OBJ  back  to 
PROP  ensures  that  visuo-spatial]  y-based  planned 
movements  are  not  implemented  without  first  tatting  into 
account  the  aims  and  objectives  of  the  user. 
Representations  at  OBJ  can,  therefore,  be  transformed  to 
factual  statements  at  PROP  and  processed  within  the 
Central  Engine  of  Cognition.  Through  comparison  with 
representations  of  the  currently  active  task,  PROP  can 
then  feed-forward  once  more  to  OBJ  which,  in  turn, 
influences  planned  movements  at  LIM.  Actual  physical 
behaviour  is,  as  a  result,  strongly  influenced  by  higher 
level  meaning  representations  and,  in  consequence,  the 
effects  upon  the  sensory  environment  then  become 
available  once  more  to  sensation,  thereby  allowing  the 
whole  process  to  begin  again. 

4.  Implications 

The  above  description  of  a  touch-based  scenario  within 
an  ICS  framework  suggests  various  implications  for  the 
development  of  effective  haptic  theory.  The  hypothesized 
processes  operating  within  the  BS  and  LIM  subsystems 
validate  the  aim  of  many  haptic  researchers  to  understand 
low-level  bodily  sensations  and  pre-motor  action  plans. 
Importantly,  however,  ICS  also  highlights  the  need  to 
consider  more  general  cognitive  processes,  including 
high-level  cognition  and  visuo-spatial  mental  imagery. 
Through  its  holistic  approach  the  theory  provides  a  means 
by  which  such  analysis  might  be  achieved.  As  described 
in  section  3  this  involves  consideration  of  the  processes 
operating  between  subsystems  across  the  overall 
architecture.  In  addition,  however,  ICS  also  postulates 
generic  principles  upon  which  representations  are 
structured  within  individual  subsystems.  In  effect, 
therefore,  although  the  types  of  data  contained  within 
each  of  the  subsystems  may  differ,  the  rules  by  which 
their  information  is  organized  will  be  the  same. 


5.  The  structure  of  mental  representations 

Representations  within  individual  subsystems  have  a 
hierarchical  structure  made  up  of  various  constituent 
items  [16].  For  example,  an  OBJ  representation  of  a  car 
dashboard  may  consist  of  the  visual  objects  ‘instrument 
display’,  ‘steering  wheel’,  and  ‘center  consol’.  Each  item, 
in  turn,  consists  of  a  group  of  further  items  one  level 
down  the  representational  structure.  In  the  present  OBJ 
example,  therefore,  the  instrument  display  may  consist  of 
‘warning  lights’,  ‘speedometer’  and  ‘revcounter’.  Indeed, 
the  hierarchical  structure  of  the  scene  may  extend  through 
many  representational  levels  down  to  the  lowest  possible 
discri  mi  liable  unit.  This  is  determined  by  the  point  at 
which  further  structural  decomposition  becomes 
impossible  due  to  a  lack  of  constituent  items  at  the  next 
level  down  the  structure. 

Within  a  representational  structure  the  item  that  is  the 
current  focus  of  attention  is  referred  to  as  the 
psychological  subject.  Through  shifts  in  attention  it  is 
possible  to  change  the  subject  to  any  item  within  a 
structure.  There  are,  however,  important  constraints  on 
the  manner  in  which  this  may  be  achieved.  For  example, 
other  items  at  the  same  level  of  structural  decomposition 
as  a  psychological  subject  form  its  context.  They  provide 
information  that  is  useful  in  discriminating  the  subject 
from  other  constituent  items  and  are  collectively  referred 
to  as  its  predicate.  It  is  possible  to  make  a  direct 
transition  of  attention  from  a  psychological  subject  to  any 
of  its  predicate  items  so  that  a  new  subject  is  formed  and 
the  old  subject  becomes  part  of  its  predicate. 

In  addition,  a  subject  and  its  predicate  form  the 
structure  of  the  next  item  above  in  the  representational 
hierarchy.  Conversely,  the  items  at  the  level  below  a 
psychological  subject  form  its  structural  group.  A  single 
transition  is  required  to  move  subject  up  or  down  a  level 
in  the  representational  hierarchy.  Shifts  of  two  or  more 
levels,  however,  require  changes  through  successive 
levels.  If,  therefore,  in  the  OBJ  representation  of  a 
dashboard,  the  psychological  subject  is  shifted  from  the 
speedometer  to  the  ‘hazard  warning  light  button’  on  the 
center  consol,  the  transition  ‘path’  must  first  move  up  a 
level  so  that  the  instrument  display  becomes  the  subject. 
The  predicate  of  the  instrument  display  will  then  include 
the  center  consol  and  so  a  further  transition  will  be 
possible  with  the  center  consol  as  the  new  subject  and  the 
instrument  display  now  part  of  its  predicate.  Finally,  as  it 
is  now  a  constituent  item  within  the  group  of  items 
forming  the  center  consol,  the  psychological  subject  can 
be  shifted  down  the  structure  again  to  the  hazard  warning 
light  button.  Importantly,  with  each  transition  there  is  an 
associated  cognitive  cost.  Complex  transition  paths 
therefore  lead  to  high  processing  demands. 
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6.  System-wide  processing 

The  OBJ  subsystem  example  is  useful  here  because  it 
permits  illustration  of  the  manner  in  which  system-wide 
cognitive  resources  may  be  involved  in  haptic  interaction. 
Say,  for  example,  that  the  user  must  investigate  our 
imaginary  dashboard  using  a  haptic  device,  with  no  other 
sensory  input.  In  addition  to  the  processing  occurring 
within  the  BS  and  LIM  subsystems,  cognitive  activity  can 
also  be  predicted  to  occur  within  other  forms  of 
representation,  including  OBJ.  Via  transformations 
through  the  level  of  meaning  (IMP  and  PROP),  visuo- 
spatial  imagery  will  be  formed  within  OBJ  with  the 
representational  structure  described  above.  This  may 
influence  further  haptic  exploration  as  the  user  attempts  to 
move  his/her  hand  in  line  with  this  ‘mind’s  eye  image’. 

The  OBJ  subsystem  example  has  been  developed  here 
purely  for  illustrative  purposes.  It  is,  however,  essential  to 
understand  that  such  structural  principles  form  the  basis 
of  processing  within  all  nine  subsystems  of  ICS.  The 
difference  between  types  of  representation  is  the  kind  of 
data  dealt  with  and  not  the  manner  in  which  that 
information  is  processed.  In  principle,  representational 
structures  can  be  hypothesized  for  all  of  the  subsystems 
involved  in  a  given  haptic  scenario  and  it  is,  therefore, 
important  for  designers  of  haptic  interfaces  to  consider 
system-wide  cognitive  processes. 

7.  Summary 

The  present  paper  deals  with  a  limited  range  of  haptic 
issues.  ICS  is  not,  however,  restricted  to  a  narrow  array  of 
applied  contexts.  Indeed,  because  the  model 
underspecifies  the  details  of  individual  types  of  mental 
processing,  it  is  possible  to  use  it  to  understand  the 
complex  cognitive  processing  occurring  within  various 
different  instances,  thereby  enabling  an  ICS  equipped 
interface  designer  to  reason  about  most  haptic  situations. 

In  addition,  it  is  possible  to  use  ICS  to  make 
theoretical  predictions  about  the  effects  of  various 
changes  to  the  interactive  situation.  For  example, 
differences  in  task,  changes  in  stimuli,  the  effects  of  other 
modalities,  user  expertise  and  knowledge,  the  availability 
of  meaningful  information,  even  the  user’s  affective  state, 
are  all  factors  that  can  be  reasoned  about  within  its 
architecture.  We  are  currently  engaged  in  further  work 
that  attempts  a  comprehensive  outline  of  ICS  within  the 
context  of  haptic  interaction,  together  with  empirical 
validation  of  the  predictions  of  the  theory  and  its  practical 
implications. 
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Abstract 

This  paper  introduces  a  system  which  converts  scanned 
line  graphs  into  a  data  format  which  is  interpretable  by  a 
haptic  device.  A  virtual  line  graph  can  then  be  rendered 
and  explored  through  touch  by  using  the  haptic  device. 
This  system  provides  an  alternative  method  for  blind  or 
visually  impaired  people  to  access  printed  graphs.  This 
method  has  a  simpler  production  process  than  the 
conventional  raised  paper.  Moreover,  exploring  graphs 
through  a  haptic  device  can  provide  assistance  and 
guidance  to  less  confident  users.  At  the  current  stage, 
simple  line  graphs  can  be  converted  and  rendered  on  two 
haptic  devices:  SensAble  PHANToM  and  Logitech 
WingMan  force  feedback  mouse.  Evaluation  of  the  system 
is  being  carried  out  and  further  improvements  have  been 
planned. 


1.  Introduction 

Automatic  conversion  of  printed  information  into 
electronic  data  formats  has  been  studied  for  many  years 
and  as  a  result,  scanners  and  Optical  Character 
Recognition  (OCR)  software  have  been  developed. 
Converting  visual  images  into  computer  recognisable  data 
provides  people  with  the  manipulation  flexibility  and  the 
ability  to  preserve  and  reprint  invaluable  documents  [1]. 
Computerising  Braille  is  one  of  the  examples  to  preserve 
aging  Braille  printout.  It  is  a  process  to  recognise  Braille 
cells  on  an  image  captured  by  a  camera.  The  electronic 
copy  of  the  Braille  document  will  be  stored  on  a  computer 
after  processing.  Although  it  is  a  good  idea  to  preserve  old 
Braille  documents  via  optical  media,  it  faces  some 
technical  difficulties,  such  as  severely  degraded  paper 
quality,  stains  and  marks  left  on  the  document  and 
problems  with  double-sided  Braille.  All  these  make  the 
recognition  difficult  and  hard  to  achieve  very  accurate 
results  [2]. 

Commercially  available  products  have  been  developed  to 
help  blind  people  to  access  printed  documents  using 
scanners,  OCR  software  and  screen  readers  [3].  The 


scanned  document  is  processed  by  the  OCR  software,  and 
subsequently  the  device  reads  out  the  recognised  text  to 
blind  people.  This  provides  blind  people  with  an  easy 
access  to  printed  documents  without  the  need  to  convert 
them  into  Braille  first.  However,  problems  of  recognising 
document  layout  and  interpreting  pictures  and  images 
cannot  be  resolved  easily  by  OCR  software.  How  many 
words  are  needed  to  describe  a  complex  graph  if  a  picture 
is  worth  a  thousand  words?  In  order  to  address  this 
problem,  we  have  been  investigating  the  possibility  of 
image  to  haptic  data  conversion  so  that  graph  features  can 
be  extracted  and  rendered  on  a  haptic  device. 

Research  work  has  been  done  on  presenting  graphical 
information  to  blind  people  using  force  feedback  devices. 
Blind  people  will  use  these  devices  to  feel  the  data 
representation  through  the  sense  of  touch  [4,  5  &  6].  Our 
previous  work  has  shown  that  blind  people  are  able  to  use 
a  PHANToM  to  comprehend  the  contents  of  line  graphs 
[7],  Therefore,  it  is  desirable  to  develop  a  system  which 
can  interpret  graphs  and  render  them  on  a  haptic  device  so 
that  blind  people  can  explore  the  data  on  the  haptic 
representation.  In  this  paper,  we  introduce  our  haptic  line 
graph  modelling  methods  and  the  attempt  to  make  image 
to  haptic  data  conversion  possible. 

2.  System 

The  system  consists  of  three  main  components:  a  flatbed 
scanner,  an  IBM  compatible  PC  and  a  force  feedback 
device  which  can  either  be  a  SensAble  PHANToM  or  a 
Logitech  WingMan  force  feedback  mouse  (Figure  1).  The 
control  flow  of  the  system  is  illustrated  on  a  schematic 
diagram  (Figure  2).  The  scanner  acts  as  a  data  input  device 
through  which  printed  graphs  will  be  acquired  and  stored 
as  digital  images.  Image  processing  techniques  are  then 
applied  to  extract  useful  features  for  the  haptic  rendering 
process.  After  the  features  have  been  extracted,  the  haptic 
rendering  will  take  place  and  the  user  can  then  explore  the 
haptic  representation  of  the  graph  through  the  force 
feedback  device. 

The  standard  TWAIN  interface  is  used  to  acquire  the 
image  of  a  printed  graph  from  an  optical  scanner. 
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Considerations  have  been  made  to  improve  the 
accessibility  of  blind  people.  TWAIN  will  call  up  the  user 
interface  provided  by  the  scanner  manufacturer  for  setting 
some  parameters  of  the  scanner  such  as  the  scanning  area, 
colour  depth  and  degree  of  exposure.  This  will  become  a 
problem  to  blind  people  as  it  is  impossible  for  them  to 
know  the  interface  arrangement  and  especially  when 
different  manufacturers  provide  different  user  interface  for 
their  scanners.  Therefore,  an  alternative  scanning  control 
is  provided.  Blind  people  can  use  a  short-cut  key  to  scan  a 
printed  graph  based  on  the  default  settings  which  would 
meet  most  needs  of  the  application. 

(a) 

Figure  1.  (a)  SensAble  PHANToM,  (b)  WingMan 
force  feedback  mouse. 


Figure  2.  Visual  to  haptic  data  conversion 
process. 

The  image  to  haptic  conversion  process  is  mainly  about 
transforming  raster  data  to  vector  representations.  The 
scanned  image  will  be  in  the  digital  form  consisting  of 
pixel  information.  The  features  that  need  to  be  exti  acted 
from  the  image  depend  on  the  requirements  of  the  haptic 
rendering  process.  Therefore,  we  need  to  know  about  the 
haptic  modelling  techniques  that  are  used  to  create  the 
haptic  graphs. 

2.1.  Haptic  Modelling 

The  haptic  modelling  of  line  graphs  varies  and  depends  on 
the  haptic  device  being  used  due  to  the  hardware 
limitations  and  software  API  support.  In  the  PHANToM 
case,  the  haptic  line  graphs  are  constructed  by  using  the 
GHOST  SDK.  Users  feel  the  line  graphs  through  the  end- 
effector  of  the  PHANToM.  The  lines  are  modelled  as 
channels  which  have  a  V-shape  cross-section  designed  to 
retain  the  PHANToM  pointer  effectively  (Figure  3).  The 
reason  for  constructing  the  line  model  as  a  concave  object 
is  due  to  the  findings  in  our  previous  study  [8].  We  found 
that  users  have  difficulties  to  keep  the  pointer  on  round 
objects  securely.  Therefore,  polygons  are  used  to  construct 
the  V-shaped  channels  and  the  inner  surface  is  defined  as 


touchable  by  the  PHANToM  so  that  the  pointer  can 
penetrate  from  the  outside  and  become  retained  in  the 
inside.  By  moving  the  pointer  along  the  channels,  users 
can  trace  the  path  of  the  lines. 


Figure  3.  Haptic  line  graph  modelling  technique 
on  the  SensAble  PHANToM. 


In  the  WingMan  force  feedback  mouse  case,  the 
Immersion  TouchSense  Developer  Toolkit  is  used  to 
create  the  force  effect.  Lines  are  modelled  by  using  the 
spring  effect  and  enhanced  by  a  friction  model.  A 
constraint  type  of  force  feedback  is  produced  by  the 
combination  of  these  two  effects.  In  operation,  when  the 
mouse  cursor  gets  close  to  the  line,  it  will  be  snapped  to 
the  line  like  entering  into  a  magnetic  field  (Figure  4). 
Strong  force  is  applied  to  the  direction  which  is  normal  to 
the  gradient  of  the  line  so  that  the  cursor  is  retained  on  the 
line  and  can  only  move  easily  along  the  path.  Sharp 
corners  are  the  source  of  problems  in  this  modelling 
technique  as  the  cursor  can  easily  shoot  off  the  line  path  if 
the  user  moves  too  fast.  A  remedy  to  this  problem  is  now 
under  investigation.  A  specific  force  feedback  at  corners  is 
being  developed  so  that  the  transition  from  one  section  of 
the  line  to  another  can  be  smoother. 


Figure  4.  Haptic  line  graph  modelling  technique 
on  the  WingMan  force  feedback  mouse. 

Despite  the  different  haptic  modelling  techniques,  the  data 
input  required  by  both  haptic  devices  is  the  same.  The 
essential  data  required  to  represent  a  line  includes  the  start, 
end  and  intermediate  points  which  are  the  turning  points 
on  the  line.  In  order  to  cope  with  multiple  lines  on  one 
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graph,  a  simple  structure  is  designed  to  allocate  the  data. 
The  information  stored  in  this  data  structure  is  the  number 
of  lines,  number  of  points  on  each  line  and  the  co¬ 
ordinates  of  each  point.  To  link  up  the  optical  system  to 
the  haptic  devices,  image  data  need  to  be  converted  into 
the  specified  data  format.  In  other  words,  pixels  of  lines 
have  to  be  detected  and  extracted  from  the  scanned  image 
and  then  transformed  into  the  data  format  required  to 
generate  the  virtual  graph. 

2.2.  Data  Conversion 

Data  conversion  relies  on  image  processing  as  useful 
information  has  to  be  extracted  and  formatted  for  the  use 
in  the  haptic  rendering.  This  is  not  an  easy  process  and 
could  be  complicated  by  several  factors,  such  as  the 
quality  of  the  printed  documents,  the  complexity  of  the 
graph  content,  and  the  type  of  graph  being  processed.  In 
the  present  development,  only  simple  line  graphs  are  used 
for  the  purpose  of  the  initial  concept  proving  study. 
Further  development  can  be  made  to  extend  the  system 
capability  to  cope  with  more  complex  contents  and  wider 
range  of  graphs. 

The  printed  graph  is  acquired  from  the  scanner  and  stored 
as  a  bitmap  file.  The  scanned  image  contains  a  large 
amount  of  information  which  is  actually  irrelevant  to  the 
haptic  rendering  process.  As  a  result,  it  needs  to  be  filtered 
out  before  the  feature  extraction  taking  place.  The 
redundant  information  includes  the  colour  on  each  pixel, 
noise  on  the  graph,  and  text  labels.  The  colour  information 
and  the  noise  can  be  reduced  in  the  initial  processing  while 
the  text  labels  will  be  ignored  in  the  feature  extraction  in 
the  later  stage.  Colour  to  greyscale  conversion  is  used  to 
reduce  the  amount  of  information  and  thresholding  is  used 
to  further  eliminate  the  unnecessary  data. 

At  this  point,  the  image  is  ready  for  the  feature  extraction 
process.  A  custom  neighbourhood  seeking  technique  is 
used  to  intemogate  the  interesting  pixel  points  on  the 
image  in  order  to  determine  whether  they  are  parts  of  a 
line  or  something  else.  It  is  based  on  convolution 
techniques  and  checks  the  surrounding  eight  pixels  for 
possible  neighbours.  If  the  point  is  a  part  of  a  line  then  the 
algorithm  will  start  tracing  the  line  and  recording  feature 
points.  The  related  pixels  are  then  allocated  together  to 
indicate  the  start,  end  and  intermediate  points  on  the  line. 
They  are  grouped  together  according  to  the  line  to  which 
they  belong  and  also  to  their  position  on  the  line. 
Therefore,  they  are  in  the  same  format  as  required  by  the 
haptic  rendering  process. 

Complications  occur  when  lines  on  the  graph  start 
intersecting  which  will  confuse  the  algorithm  as  it  is 
generally  difficult  to  know  whether  the  points  belong  to 
the  original  line  or  another  one.  Some  assumptions  are 
needed  to  deal  with  this  kind  of  situation,  i.e.  assuming 
lines  would  not  change  direction  at  the  intersection  and 
continue  on  the  same  gradient.  More  investigations  are 
required  to  explore  different  conditions  and  obtain 
satisfactory  solutions. 


3.  Future  Work 

In  the  current  development,  the  system  only  deals  with 
simple  line  graphs  however  the  concept  of  using  image  to 
haptic  data  conversion  to  provide  access  to  printed  graphs 
to  blind  people  is  demonstrated.  It  is  possible  to  tackle 
more  complex  line  graphs  such  as  multiple  intersecting 
lines  and  lines  with  different  patterns.  Moreover, 
converting  other  types  of  graphs,  such  as  bar  charts,  pie 
charts  and  other  types  of  data  visualisation  techniques,  will 
be  investigated  and  supported  in  the  system.  Besides 
expanding  the  capability  of  the  system,  an  orientation 
adjustment  method  is  under  investigation  so  that  the  right 
orientation  of  the  scanned  graph  can  be  automatically 
detected.  This  will  improve  blind  people’s  accessibility  in 
the  system  as  it  is  difficult  for  them  to  tell  the  orientation 
of  a  printed  document  without  any  form  of  assistance. 
Furthermore,  evaluation  of  the  system  is  being  carried  out. 
The  main  investigations  are  focused  on  two  different 
aspects:  accessibility  and  usability  of  the  system,  and  the 
effectiveness  of  the  feature  extraction  techniques 
developed  in  the  system. 
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Abstract 

We  report  an  experiment  in  which  people 
were  asked  to  make  a  judgement  about  the 
gradient  of  a  simple  line  graph.  As  part  of  the 
MultiVis  project  on  data  visualisation  for  blind 
people  our  aim  was  to  discover  how  accurately 
linear  graphs  can  be  rendered  using  relatively 
simple  technology:  raised  paper,  and  a  haptic 
force-feedback  mouse.  Results  show  that  both 
media  allowed  very  accurate  performance. 


Introduction 

This  work  is  part  of  the  MultiVis  project 
(multivisualisation  for  the  blind).  The  main 
objective  of  this  project  is  to  make  statistical 
information,  such  as  graphs  and  tables,  more 
accessible  to  blind  people.  Using  virtual  reality 
technologies,  it  is  hoped  this  will  be  achieved 
by  using  alternative  forms  of  displaying 
information  to  the  blind  and  the  visually 
impaired. 

Blind  and  visually  impaired  people  interact 
with  the  world  using  auditory  and  tactile 
sensory'  modalities.  For  this  reason,  the  project 
explores  the  use  of  3D  sound  and  virtual 
reality  haptic  presentation  to  render 
information.  The  focus  of  this  study  was  the 
haptic  domain,  and  in  particular  its  utility  for 
presenting  information  related  to  virtual  reality 
line  graph  displays. 

The  Study 

The  aim  of  this  study  was  to  examine  how' 
accurate  people  are  at  processing  varying 
gradients  of  simple  line  graphs  presented  as 
tactual  lines  on  raised  paper  or  as  virtual  haptic 
lines  via  a  force-feedback  mouse. 

A  comparison  between  physical  and  virtual 
reality  media  will  allow  assessment  of  the 
haptic  force-feedback  device.  In  particular,  we 
were  interested  to  establish  whether  the  force- 
feedback  mouse  would  provide  an  accurate 


means  for  presenting  simple  linear  graphs 
when  compared  to  physical  raised  lines  (on  a 
relief  map).  If  it  is  accurate,  the  virtual  reality 
device  has  many  advantages  over  the  physical 
map,  particularly  in  the  fact  that  the  display 
can  be  changed  very'  simply  (e.g.,  by  re¬ 
rendering  information)  whereas  separate 
physical  displays  (relief  maps)  need  to  be 
constructed  for  each  new  presentation. 

From  the  psychological  perspective,  there 
has  been  little  research  on  the  processing  of 
tactile  line  orientation  [eg  1, 3],  The  patterns  of 
results  obtained  from  these  studies  generally 
focus  on  particular  phenomena  such  as  the 
"haptic  oblique  effect".  This  show's  poorer 
performance  in  oblique  linear  orientations  than 
in  vertical  and  horizontal  ones  [1].  These 
experiments  have  been  conducted  using 
physical  media  that  can  be  fully  explored  by 
touch.  Whether  haptic  perception  of  virtual 
line  orientations  follows  the  same  pattern  as 
w  ith  physical  media  is  an  important  question  in 
terms  of  data  visualisation  for  the  blind  [4,  5], 
It  is  by  no  means  clear  that  the  two  media  will 
be  equivalent  in  this  regard.  Virtual  reality 
force-feedback  devices  are  limited  to  a  single 
point  of  contact,  and  do  not  support  the  direct 
skin  contact  available  for  physical  stimuli.  If 
virtual  reality  force-feedback  media  are  to  be 
used  to  render  line  graphs  for  blind  people,  it  is 
important  to  establish  whether  these  media  are 
sufficiently  sensitive  to  do  so.  Since  line 
orientations  in  a  line  graph  are  a  primary 
source  of  information,  we  began  our 
investigations  by  asking  whether  people  can 
process  the  gradient  of  a  given  line  with  such 
media.  To  investigate  this  question,  an 
experiment  was  set  up  which  used  simple 
linear  graphs  and  a  forced-choice  paradigm. 

Experiment 

20  participants  were  recruited  for  this 
experiment.  All  participants  were  sighted,  as 
well  as  right-handed.  None  had  any  known 
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sensory  or  motor  disabilities  which  might  have 

affected  their  haptic  perception. 

A  set  of  virtual,  haptic  stimuli  was 
presented  using  Logitech’s  WingMan  force- 
feedback  mouse.  The  virtual  stimuli  for  use 
with  this  mouse  were  generated  using  the 
Immersion  Studio  Application  (version  3.4.2) 
software. 

Another  set  of  stimuli  was  presented  on  raised 
paper  (swell  paper).  This  allows  the 
photocopying  of  stimuli  onto  special  paper, 
which  is  then  put  through  a  HotSpot  machine 
raising  the  darkened  areas  through  a  heating 
process,  and  producing  a  relief  map. 

For  the  virtual,  haptic  stimuli  a  set  of  68  virtual 
lines  was  generated.  These  consisted  of  four 
replications  each  of  a  set  of  angled,  virtual 
lines  taking  a  horizontal  orientation  (0°)  as 
reference  point.  This  formed  the  basis  for 
producing  virtual  lines  at  stepwise  angular 
increases  of  3°  each,  both  sloping  up  and  down 
from  the  horizontal,  up  to  24°  cither  side  of  the 
horizontal.  This  generated  a  set  of  virtual, 
angled  lines  at  the  following  angles  (taken 
from  the  horizontal):  3°,  6°,  9°,  12°,  15°,  18°, 
21°,  24°  (all  sloping  up),  -3°,  -6°,  -9°,  -12°,  - 
15°,  -18°,  -21°,  -24°  (all  sloping  down)  and  0° 
(horizontal).  Furthermore,  four  virtual  haptic 
practice  lines  were  generated  consisting  of  2  x 
30°  (upwards)  and  2  x  -30°  (downwards) 
angled  lines. 

Generating  the  virtual  lines  themselves  was 
accomplished  using  the  Enclosure  Effect 
within  the  Position  Based  Effect  which  is 
supplied  with  the  Immersion  Studio  software. 
This  Enclosure  Effect  gives  the  impression  of  a 
haptic  line  which  is  determined  by  four  walls 
as  boundaries.  In  this  way,  participants 
hapttcally  perceive  a  line  as  an  engraved 
groove.  Pilot  studies  had  shown  that  an 
"engraved"  effect  was  easier  for  subjects  to  use 
than  a  "raised"  effect.  The  latter,  though 
approximating  more  closely  tc  a  raised-paper 
line,  tended  to  allow  subjects  to  lose  the  region 
of  interest  by  "falling  off'  the  line  [5], 
Stiffness  and  saturation  for  all  stimuli  was  set 
to  10,000  units  for  lefl/right  and  top/bottom 
walls.  Thickness  of  these  was  set  to  25  units.  A 
stiffness  mask  was  activated  for  each  stimulus 
pertaining  to  Tn_Top’,  ‘InBottom’,  Tn_Left’ 
and  Tn_ Right’.  The  centre  width  of  each  line 
(ie  enclosure)  was  set  to  500  units  for  width 
and  50  units  for  height.  For  each  virtual  line 
boundary  coordinates  were  variable  as  were 
the  centre  settings  for  both  the  x  and  y 
coordinates.  All  lines  were  presented  to 
participants  in  the  centre  of  the  haptic  display 
field  of  the  force-feedback:  mouse.  The 
strength  of  the  force-feedback  was  set  to 
maximum  strength  throughout. 


The  haptic  ‘viewing’  field  in  which  the  force- 
feedback  mouse  operates  was  4cm  x  3.5cm  in 
size. 

For  the  raised  paper  lines  the  same  set  of  68 
angled  lines  and  four  practice  lines  as  for  the 
virtual  stimuli  was  created.  Lines  for  these 
stimuli  measured  9.7cm  in  length  and  .5cm  in 
width.  Raised  line  stimuli  were  presented  to 
participants  in  a  styropor  foam  frame.  This 
allowed  participants  to  feel  the  stimuli 
themselves,  as  well  as  the  boundaries  set  by 
the  frame.  The  overall  frame  size  measured 
37.5cm  x  25.5cm.  The  haptic  ‘viewing’  field 
measured  13.5cm  x  9.8cm. 

Design  and  Procedure 

There  were  two  conditions  in  this 
experiment:  raised  lines  and  virtual  lines.  The 
presentation  of  the  two  conditions  was  blocked 
and  the  order  of  presentation  for  these 
counterbalanced.  Participants  were  randomly 
allocated  to  the  presentation  order  of  the  two 
conditions  and  were  blindfolded  throughout 
the  experiment.  They  were  presented  with  the 
linear  graphs,  and  asked  to  say  whether  the 
gradient  was  positive  or  negative  (sloping 
upwards  or  downwards  from  left  to  right). 
They  were  instructed  to  guess  if  they  were 
unsure  unsure  of  the  direction  of  a  gradient. 

For  each  virtual  line  the  mouse’s  cursor  was 
positioned  on  the  line’s  centre.  The  mouse 
itself  was  positioned  straight  on  its  fixed  pad 
and  placed  in  front  of  participants.  In  the 
virtual  lines  condition  participants  were  firstly 
familiarised  with  the  mouse  and  its  force- 
feedback  by  letting  them  explore  two 
demonstration  files  supplied  with  the 
WingMan  utilities,  FEELitObjects. 

For  the  raised  paper  lines,  the  styropor  frame 
was  fixed  in  front  of  participants. 

Once  a  given  line  was  set  up  for  presentation, 
participants  moved  their  own  hand  into 
position  to  feel  it.  In  order  to  reflect  real  world 
users’  variability  in  exploring  objects  tactually, 
participants  were  allowed  to  explore  the  lines 
in  any  manner  they  wished,  employing  any 
strategy  they  wanted.  They  were  also  allowed 
to  explore  each  line  without  time  limit. 
Participants  were  given  four  practice  stimuli 
for  each  condition. 

Results 

The  percentage  of  correct  responses  for 
both  the  raised  lines  (swell  paper)  and  virtual 
lines  (force-feedback  mouse)  can  be  seen  in 
Figure  1. 
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Figure  1.  Participants’  responses  to 
raised  and  virtual  lines. 


Statistical  analysis  was  conducted  on  data 
from  -9  to  +9  degrees,  as  inspection  showed 
ceiling  effects  beyond  this  range.  Within  this 
range,  there  was  a  very  large  effect  of  angle, 
reflecting  a  drop  in  performance  as  the  angle 
became  shallower  (F6,  114)  =  10.5;  p  <  0.05). 
There  was  no  significant  interaction  (F(6,  114) 
<  1)  reflecting  the  fact  that  decreasing  angles 
affected  both  presentation  media  equally. 

In  the  case  of  zero  degrees,  participants  did 
not  behave  at  random,  but  exhibited  a  bias 
towards  responding  “downwards”.  This  bias 
was  significant  across  the  groups  (single 
sample  t-test,  t(19)  =  4.3;  p  <  0.05)  and 
equivalent  for  each. 

These  results  show  that  both  media  allowed 
highly  accurate  performance,  and  that  the 
pattern  of  errors  appears  to  be  equivalent 
across  each. 

Discussion  and  Conclusion 

First,  we  note  that  subjects  were  very 
accurate  at  this  task.  Within  roughly  6  degrees 
of  the  horizontal  they  perform  at  ceiling  levels. 
Even  within  3  degrees  of  the  horizontal, 
performance  is  very  good.  Second,  we  note 
that  the  pattern  of  accuracy  and  errors  appears 
to  be  very  similar  across  the  two  media.  This  is 
perhaps  surprising,  because  the  physical 
raised-paper  appears  to  off  richer  information. 
Interacting  with  a  physical  object,  subjects  can 
use  both  tactile  and  haptic  cues,  whereas  the 
force  feedback  mouse  allows  only  a  point- 
haptic  interaction. 


The  pattern  of  results  observed  in  this  stud)' 
supports  the  use  of  virtual  reality  haptic  media 
in  the  rendering  of  line  graphs  for  blind  people. 
Although  employing  blindfolded  participants, 
there  is  no  reason  to  suggest  the  pattern  of 
results  should  be  different  for  blind  people. 
The  basic  psychological  processes  required  to 
make  gradient  decisions  on  differently  sloping 
simple  line  graphs  should  be  the  same  for 
seeing  and  blind  people. 

Furthermore,  the  degree  of  accuracy  in  making 
gradient  decisions  in  both  media  is 
encouraging  in  regards  to  the  use  of  this  form 
of  information  rendering  for  statistical  data 
such  as  line  graphs  for  the  blind. 

An  experiment  is  currently  in  progress  to 
expand  the  present  stud)  to  the  use  of  the 
PHANToM  force-feedback  device.  This 
experiment  investigates  whether  the 
orientation  of  the  presentation  plane  (ie 
horizontal  vs  vertical)  has  an  effect  on 
perception  of  line  gradient,  since  each  plane 
presents  a  different  context  of  gravitational 
cues  [2], 

We  conclude  that  the  haptic  mouse  allows 
very  accurate  rendering  of  simple  line  graphs. 
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Abstract 

One  of  the  possible  actuation  principles  which  were 
found  to  be  the  most  promising  candidates  for  high- 
resolution  tactile  displays  is  presented.  The  actuators  are 
based  on  SMA  wire,  which  includes  a  clamping 
mechanism,  to  overcome  high  energy  consumption  and  its 
consequences.  A  module  consisting  of  16  taxels  in  one 
row,  with  a  distance  of  2.54mm  separating  the  taxels,  was 
designed  and  produced  with  jjEDM  technology.  Special 
setting  and  resetting  procedures  have  been  developed 
with  the  purpose  of  overcoming  friction  problems  that 
were  noticed  during  the  first  tests.  The  design  and  the 
procedures  are  presented. 


1.  Introduction 

Tactile  displays  are  already  available  on  the  market, 
but  only  for  Braille  text  (mostly  in  one  row).  The 
conventional  actuators  used  in  such  commercial  products 
(piezoelectric  bending  elements)  are  not  suitable  for  high- 
resolution  graphic  tactile  displays  because  of  their 
dimensions  and  high  price. 

Fast  development  of  micro-manufacturing 
technologies,  as  well  as  the  micro-actuators  produced  with 
them,  has  opened  new  possibilities  for  the  production  of 
actuators  which  are  suitable  for  meeting  the  requirements 
of  a  high-resolution  graphic  tactile  display. 

After  evaluation  of  the  available  actuation  principles, 
some  of  the  most  promising  candidates  were  chosen  for 
further  testing  and  development.  The  evaluation  criteria 
employed  allow  full  compatibility  with  standard  Braille 
text  and  provide  good  representation  of  the  graphic  items. 
The  main  criteria  employed  for  evaluating  both  the 
actuation  principles  and  the  actuators  designed  according 
to  these  principles  were  as  follows: 

_ _ Table  1 .  Requirements _ 

•  Possibility  of  miniaturisation  (the  actuator  has  to  allow 
packaging  of  at  least  10  actuators  per  inch,  i.e.  the 
outer  dimension  of  a  single  actuator  must  not  exceed 
2.54mm).  _ 


•  Amplitude  of  vertical  movement  has  to  be  0.6  - 
0.8mm. 

•  Actuator  should  be  able  to  bear  a  vertical  force  of  at 
least  lOOmN  (an  optimum  load  would  be  150mN). 

•  Possibility  of  integrated  production  with  minimal 
additional  assembling. 

•  Possibility  of  independent  manipulation  without 
mutual  influence  of  neighbouring  actuators. 

•  The  response  speed  has  to  allow  setting  and  resetting 
of  the  display  within  a  reasonable  amount  of  time  (on 
the  order  of  seconds). 

•  Good  long-term  stability. 

•  Low  cost. 


2.  Principles 

There  are  several  basic  principles  that  can  be  used  to  form 
a  convexity  on  a  surface.  They  can  be  divided  into  main 
groups  as  given  below: 

_ Table  2.1.  Principles _ 


Linear  movement 

isb: 

Rotation 

— - 

Table  2.2.  Principles 
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Change  of  the  length 

1  - 

|  1+  Al 

Change  of  volume 

.»  -1 

1  V+  AV 

Change  of  shape 

$  — v 

After  the  evaluation  of  possible  solutions  (see  tables  2 
and  3),  the  following  principles  showed  that  they  are 
suitable  for  achieving  the  requested  criteria: 

-  Actuator  based  on  SMA  wire  which  uses  a  clamping 
system  to  prevent  high  energy  consumption  and  its 
consequences. 

-  A  valve  matrix  based  on  a  fluid  that  can  change  its 
viscosity  under  the  influence  of  an  electric  field  (electro- 
rheological  fluids  or  liquid  crystals);  this  matrix  uses  an 
external  fluid  pump  to  provide  a  continuous  fluid  flow  and 
the  required  pressure. 

-  A  matrix  of  mechanically  multiplexed  micro-valves 
based  on  sequential  driving  (time  multiplex)  row  by  row. 
The  advantage  of  that  system  is  that  it  reduces  number  of 
actuators  necessary  (for  the  matrix  consisting  m  x  n 
valves,  the  number  of  the  actuators  is  m  +  n). 

The  actuator  based  on  SMA  wire  will  be  presented  in 
this  paper,  while  the  other  two  principles  are  still  in  the 
phase  of  investigation. 

2.  The  actuator  based  on  SMA  wire 

Different  actuator  designs  using  SMAs  (Shape 
Memory  Alloys)  intended  for  tactile  displays  have  already 
been  presented  (see  references  [1],  [2]). 

Table  3.  Principles  and  solutions 
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The  advantages  of  the  SMA  actuation  principle  are 
simplicity,  compactness,  and  safety  of  the  mechanism,  a 
high  power/weight  ratio,  and  clean  and  silent  operation. 
The  SMAs  are  mostly  used  as  springs,  though  also  in  the 
form  of  plates  and  wires  of  various  diameters.  In  the  case 
of  SMA  wires,  the  ratio  of  the  force  that  can  be  delivered 
to  the  wire’s  cross-section  is  especially  high.  There  is  no 
other  actuation  principle  which  can  deliver  350mN  of 
force  with  a  cross-section  of  only  50pm  (the  parameters  of 
the  wire  used  in  the  presented  actuator). 

The  main  disadvantage  of  the  Shape  Memory  Alloys’ 
actuating  principle  is  low  efficiency  (about  5%),  and 
consequently  high  thermal  dissipation.  For  the  presented 
actuator,  thermal  dissipation  on  the  wire  is  approximately 
0.06W  (when  active)  per  actuator.  A  larger  matrix  of 
those  actuators  (an  ideal  graphic  tactile  display  is  usually 
defined  as  a  matrix  consisting  of  320  x  240  taxels)  would 
have  a  thermal  dissipation  on  the  order  of  kilowatts. 
Despite  the  high  energy  consumption  of  the  display  as  a 
whole,  more  important  problems  are  the  high  density  of 
the  thermal  energy  and,  as  a  consequence,  the  high  mutual 
influence  of  neighbouring  actuators.  It  would  be 
practically  impossible  to  keep  one  actuator  inactive  when 
it  is  surrounded  by  a  number  of  active  ones  for  longer  than 
several  seconds. 

If  we  consider  that  one  “reading”  of  some  graphic  (or 
several  rows  of  textual)  tactile  information  can  take  tenths 
of  a  second  or  more,  it  can  be  concluded  that,  instead  of 
actuators  that  require  energy  in  the  active  state,  the 
actuators  working  in  “bi-stable”  mode  are  more 
appropriate.  These  actuators  require  energy  only  when 
switching  to  one  of  two  possible  states.  No  energy  is 
required  in  any  of  two  end  positions. 

One  of  the  ways  of  achieving  that  effect  is  to  use  an 
appropriate  clamping  mechanism  which  holds  the  actuator 
in  position  until  the  change  is  necessary. 

The  actuator  using  SMA  wire  as  an  active  element  and 
a  micro-machined  clamping  mechanism  was  designed  and 
produced  at  the  Institute  for  Micro  Technique  and 
Precision  Engineering.  The  functioning  principle  is 
presented  in  Fig.  1 . , 

An  SMA  wire  works  in  the  following  manner:  At 
temperatures  below  the  transition  temperature,  the  wire 
can  be  easily  extended  using  a  weak  force  (bias  force).  If 
the  wire  is  heated  above  the  transition  temperature  (the 
easiest  way  to  heat  it  is  to  apply  an  appropriate  electrical 
current  directly  to  the  ends  of  the  wire)  it  contracts  till  it 
reaches  the  previously  programmed  length.  During  the 
contraction,  the  wire  can  deliver  a  force  that  is  5-10  times 
stronger  than  the  force  necessary  for  extension. 

SMA  alloy  wire  (trade  name  FLEXINOL,  produced  by 
DYNALLOY,  Inc.)  with  a  diameter  of  50pm  and 
transition  temperature  of  70°C  is  used  as  an  active 
element.  This  wire  can  provide  a  force  of  about  350mN 


Figure  1 .  Principle  of  SMA  actuators  and  their  clamping 
mechanism 

during  contraction.  If  extension  of  the  wire  does  not 
exceed  3%  of  its  active  length,  it  could  provide  hundreds 
of  thousands  of  actuations  (data  given  by  the  producer) 
without  losing  its  initial  properties.  Long-term  stability 
tests  have  not  been  done  yet. 

The  active  length  of  the  wire  used  was  30mm  (not 
whole  lengths  as  presented  in  Fig.  1),  while  maximal 
displacement  was  limited  to  0.8mm.  This  results  in  a  wire 
extension  of  2.66%,  which  is  below  the  critical  limit. 

Elastic  springs  are  used  to  provide  the  necessary  bias 
force,  and  they  are  chosen  so  that  they  can  bear  a  vertical 
load  of  150mN  in  the  upper  position  (the  force  required 
for  standard  Braille  actuators). 

The  upper  end  of  the  wire  is  fixed  to  the  taxel,  and  its 
lower  end  is  fixed  to  the  contact  pin  by  means  of 
mechanical  clamping. 

The  main  part  that  holds  all  of  the  parts  serves  as  a 
common  electrical  contact  (all  of  the  parts  are  made  of 
metal),  while  contact  pins  (isolated  from  the  main  part) 
fixed  to  the  lower  end  of  each  wire  provide  an  electrical 
contact  for  each  single  actuator,  as  well  as  the  releasing 
(resetting)  actuator. 

In  the  starting  (reset)  state,  the  actuators  are  in  the 
upper  position  (on  the  left  in  Fig.  1).  In  this  position, 
clamping  springs  are  stressed,  providing  a  very  weak  force 
that  presses  the  corresponding  taxel  from  the  left  to  the 
right  side.  The  force  is  so  weak  that  it  is  unable  to  move 
the  taxel.  If  an  appropriate  electrical  current  (the  current 
of  90mA  was  found  to  be  the  best  compromise  regarding 
fast  response  and  high  consumption)  is  applied  between 
the  main  part  and  one  or  more  contact  pin(s), 
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corresponding  SMA  wires  are  heated,  and  they  contract 
until  the  predefined  length  is  reached.  The  actuators  are 
adjusted  so  that  the  corresponding  clamping  spring  is 
released  shortly  (about  200pm)  before  the  taxel  reaches 
the  lowest  position.  The  electrical  current  is  than 
disconnected,  and  the  elastic  spring  tends  to  extend  the 
SMA  wire. 

However,  the  taxel  is  clamped  in  the  lower  position 
under  the  influence  of  the  clamping  spring,  which  is  now 
located  next  to  the  right  taxel  in  Fig.  1.  No  energy  is 
required  to  keep  the  taxel  in  this  position  and  its  upper 
position. 

When  the  taxels  have  to  be  reset,  a  slider,  also 
presented  in  Fig.  1,  is  pulled  from  right  to  left.  The 
clamping  springs  are  pulled  to  the  taxel  at  the  same  time 
so  that  elastic  spring  can  extend  the  SMA  wire  to  the 
upper  position. 

According  to  the  described  principle,  a  module  was 
designed  and  produced;  it  consists  of  16  taxels  in  one  row, 
separated  by  2.54mm,  as  well  as  one  releasing  actuator 
which  also  uses  SMA  wire  (two  parallel  150pm  wires 
instead  of  one  50pm  wire  are  used  for  each  taxel)  and 
resets  all  16  taxels  at  once. 

The  main  part,  which  holds  all  of  the  other  parts,  the 
slider,  the  rotating  part  that  serves  as  a  lever  mechanism 
for  the  slider  (see  Fig.  4),  and  the  clamping  springs,  was 
produced  using  pEDM  technology  [3],  The  taxels  and 
contact  pins  were  milled. 

REM  pictures  of  some  of  the  important  details  are 
presented  in  Fig.  2,  Fig.  3  and  Fig.  4. 


Figure  2.  REM  picture  showing  three  of  the  taxels,  as  well 
as  the  position  of  the  clamping  springs 


Figure  3.  REM  picture  that  shows  the  design  and  position 
of  the  slider,  which  serves  to  pull  the  clamping  springs, 
thereby  releasing  the  taxels 


Figure  4.  REM  picture  that  shows  the  design  of  the  lever 
mechanism,  which  serves  to  transfer  the  vertical 
movement  produced  by  the  SMA  wire  and  the  elastic 
spring  accompanying  it  to  the  horizontal  movement  of  the 
slider 

The  main  problem  that  appeared  in  the  first  tests  of  the 
system  was  the  friction  between  the  slider  and  both  the 
main  part  and  the  taxels  that  were  in  the  upper  position 
during  resetting.  To  overcome  this  problem,  a  resetting 
procedure  was  developed.  The  time  diagrams  of  the 
setting  and  resetting  procedures  are  presented  in  Fig.  5. 

The  time  diagrams  show  the  control  voltages  applied  to 
the  drivers  that  provide  the  required  electrical  current  to 
the  SMA  wires.  LOW  logical  levels  are  active  states  (the 
current  flows  through  the  wire).  The  “line  enable”  signal 
is  used  because  the  module  is  intended  to  be  part  of  a 
matrix.  The  “line  enable”  signal  actually  addresses  the 
selected  module  (multiplex  driving  will  be  used). 

To  set  the  selected  taxels,  no  special  procedure  is 
necessary.  The  time  necessary  to  move  one  taxel  from  the 
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upper  to  the  lowest  position  was  determined  to  be  0.5 -0.7s 
(if  a  current  of  90mA  is  applied).  To  be  sure  that  all  of  the 
selected  taxels  have  reached  their  lowest  position,  the 
current  must  flow  through  the  SMA  wire  (Tset  in  Fig.  5) 
for  a  longer  amount  of  time.  The  period  of  Ty..=ls  was 
found  to  be  the  most  appropriate. 


T3=0.2s 

Step  3:  After  the  period  of  Tl-T3=0.3s,  the  taxels  are 
deactivated,  but  the  slider  continues  to  pull  till  the  taxels 
have  reached  upper  position. 

T4=1.5s 


SET  PROCEDURE: 
TO 


RESET  PROCEDURE: 


Step  4:  The  slider  is  now  deactivated,  but  some  time  is 
necessary  till  the  corresponding  SMA  wire  cools  down 
and  relaxes. 

T5=1.5s 

Step  5:  As  the  bias  spring  is  not  strong  enough  to  push 
the  slider  to  the  start  position  (the  slider  is  again  clamped 
between  the  taxels  and  the  main  part),  a  short  pulse  must 
be  applied  to  all  of  the  taxels.  The  slider  is  released  for  the 
short  period,  which  is  enough  for  it  to  reach  its  start 
position. 

T2=0.2s 


Figure  5.  Time  diagrams  of  the  control  voltages  during 
setting  and  resetting  of  the  taxels 

Concerning  resetting,  several  consequent  steps  are 
necessary.  The  steps  and  the  corresponding  time  periods 
(see  Fig.  5)  are  as  follows: 

Step  1:  All  of  the  taxels  are  activated  (the  current  starts 
to  flow  through  the  corresponding  SMA  wires).  This 
releases  the  slider,  which  is  no  longer  clamped  between 
the  taxels  and  the  main  part,  and  it  can  now  slide  freely 
without  friction.  It  also  helps  that  the  clamping  springs  are 
not  pressed  by  the  taxels  from  below,  so  that  only  their 
elastic  forces  are  present  when  the  slider  is  being  pulled. 

Tl=0.5s 

Step  2:  Shortly  after  the  taxels  are  activated,  the  slider 
is  activated,  and  it  starts  to  pull  the  clamping  springs. 


The  whole  resetting  time  is: 

Tresel=T3+T4+T5+T2=3.4s 

If  a  higher  current  were  applied,  both  the  setting  and 
resetting  times  would  be  shorter,  but  this  could  influence 
the  long-term  stability  of  the  system  as  a  whole. 

3.  Conclusion 

To  a  design  of  micro-actuators  based  on  SMA  wire 
which  uses  an  appropriate  clamping  mechanism  was 
presented.  The  actuators  were  initially  intended  for 
graphic  tactile  displays  for  the  blind,  but  they  could  be 
used  wherever  tactile  feedback  is  desired  (in  minimally 
invasive  surgery  ,  by  fighter  jet  pilots,  etc.). 

Further  evaluation  as  well  as  the  necessary 
improvements  of  the  presented  actuators  will  be  done. 
Improvement  of  the  surface  quality,  either  by  using  a 
different  manufacturing  technology  or  polishing  the 
surfaces,  could  solve  the  friction  problems  and  allow 
simpler  (faster)  resetting.  The  next  step  would  be  adapting 
the  presented  design  for  production  with  microinjection 
moulding,  which  would  reduce  the  price  for  serial 
production.  Some  ways  of  realizing  easier  (automated) 
assembly  and  adjustment  of  the  system  have  already  been 
discussed  by  the  working  group.  And  finally,  though  not 
less  important,  is  finding  a  possibility  to  integrate  a 
feedback  mechanism  with  the  purpose  of  providing  some 
degree  of  interactivity. 

The  work  concerning  the  other  mentioned  principles  is 
also  ongoing. 
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Figure  6.  Assembled  module  consisting  of  16  actuators  in 
one  row 


Figure  7.  Experimental  set  up 
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Abstract 

This  paper  describes  the  potential  of  using  vibro-tactile 
displays  for  automobile  drivers.  Technological 
developments  in  the  field  of  driver  support  systems  and 
tactile  displays,  combined  with  the  ever  increasing  need  to 
enlarge  the  capacity  of  the  driver’s  information  channel, 
form  the  reason  to  review  the  possibilities  of  in-car  tactile 
displays  and  to  identify  some  promising  applications.  In  the 
second  part  of  the  paper,  we  describe  a  feasibility  study  in 
which  we  tested  an  in-car  tactile  display  in  a  driving 
simulator.  The  results  show  that  the  tactile  navigation 
display  resulted  in  better  performance  compared  to  a  visual 
display,  and  that  it  reduces  the  driver’s  workload.  This 
study  gives  a  first  indication  that  employing  the  tactile 
modality  may  be  a  major  step  to  accomplish  safety 
improvements. 

1.  Motivation 

In-car  systems  can  have  a  negative  effect  on  safety  when 
they  enlarge  the  workload  or  distract  the  driver  [1].  The 
interaction  of  the  driver  with  the  in-car  system  may  result  in 
an  overload  situation  in  which  the  driver  no  longer  processes 
all  the  information  relevant  for  the  driving  task,  which  may- 
result  in  errors  and  late  detection  of  other  road  users  [2]. 
Based  on  the  assumptions  of  the  multiple  resource  model  [3] 
which  predicts  no  performance  degradation  when 
independent  resources  are  used,  and  the  fact  that  most 
information  in  driving  is  visual,  safety  maybe  improved  by 
presenting  information  via  audition  and  touch  instead  of 
vision. 

1.1.  Vibro-tactile  perception  and  displays 

Vibro-tactile  displays  in  general  consist  of  arrays  of 
vibrating  elements  coupled  to  the  skin.  Appropriate 
stimulation  of  specific  receptors  in  the  skin  by  means  of 
applying  localised  vibration  typically  leads  to  a  ’tickling’ 


sensation  at  that  specific  location.  Human  vibro-tactile 
perception  is  sensitive  to  location,  duration,  frequency, 
amplitude  and  several  other  (derived)  aspects  of  vibro-tactile 
stimulation.  Obviously,  the  Braille  reading  skills  of  visually 
impaired  people  have  proven  our  general  ability  to  process 
complex  spatio-temporal  patterns  presented  to  our  skin. 

1.2.  The  tactile  modality  in  man-machine-interfaces 

State  of  the  art  man-machine-interfaces  (MMIs)  almost 
routinely  use  the  visual  and  auditive  channel  only.  Using  the 
haptic  and  tactile  channel  in  MMIs  is  rather  uncommon, 
despite  good  reasons  to  do  so.  Advantages  of  the  tactile 
channel  are,  amongst  others,  that  it  is  always  ready  to  receive 
information,  that  it  draws  attention,  that  it  is  private,  and  that 
it  can  be  used  in  a  natural  and  intuitive  way.  The  latter  is  of 
course  dependent  on  the  kind  of  information  that  is 
presented.  Examples  include  displaying  object  properties 
such  as  roughness  [4] ,  and  variants  of  a  fap-on-the-shoulder’ 
to  present  navigation  information.  Another  good  reason  to 
incorporate  the  vibro-tactile  channel  in  MMIs  is  that  adding 
the  tactile  modality  may  enlarge  the  total  effective 
information  processing  capacity  of  the  user.  It  may  thus  free 
the  overloaded  auditory  and  /or  visual  modalities. 

Recent  developments  resulted  in  small,  cheap,  robust, 
and  readily  available  tactile  display  elements  (e.g.,  those 
used  in  mobile  phones  and  pagers).  These  simple  elements 
can  be  combined  into  sophisticated  displays.  This  will 
eventually  result  in  the  introduction  of  relative  simple  tactile 
displays  which  have  a  much  larger  information  transfer 
capacity  than  the  1-bit  your  phone  is  ringing’  message. 
Recent  applications  of  these  vibro-tactile  displays  hint  at 
more  intuitive  ways  of  using  the  skin  as  an  information 
channel  [5].  For  instance,  it  has  been  shown  [6]  that 
stimulating  a  single  point  on  the  torso  immediately  leads  to  a 
perception  of  external  direction.  Such  an  intuitive  perception 
of  external  direction  can  be  very  useful  in  applications 
requiring  a  sense  of  spatial  awareness,  such  as  when 
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navigating  a  car  through  an  environment,  operating  in  space 
[7]  or  when  flying  super-agile  aircraft  [8,  9]. 

1.3.  Classes  of  information  in  car  driving 

After  recognising  the  potential  of  applying  tactile  interfaces 
in  an  automobile,  dimensions  that  need  to  be  studied  are  the 
types  of  information  and  the  coding  possibilities  that  are 
suitable.  We  discern  the  following  classes  of  information: 

a)  Spatial  information.  Spatial  information  may  be  one  of 
the  most  promising  areas  of  applying  tactile  displays  [9]. 
First  of  all,  the  visual  system  is  restricted  in  field-of-view. 
This  means  that  it  normally  extracts  spatial  information  from 
the  1 80°  forward  view  only.  Not  perceivi ng  relevant  objects 
in  space  (e.g.,  cars  in  dead  angles,  crossing  pedestrians)  is  a 
major  cause  of  accidents.  Secondly,  presenting  3D 
information  on  a  2D  visual  display  may  result  in  errors  and  a 
high  visual  workload.  Presenting  such  information  to  the 
skin  is  likely  to  reduce  those  problems. 

b)  warning  signals.  Since  the  tactile  channel  is  always 
ready  to  receive  information  and  has  the  characteristic  of 
drawing  the  observer’s  attention,  it  is  well  suited  for 
warnings.  Furthermore,  a  visual  warning  on  the  dashboard 
may  be  on  for  a  long  time  before  the  driver  actually  notices 
it,  and  auditive  warning  signals  can  be  lost  in  tyre,  radio,  or 
conversation  noise. 

c)  communication.  Tactile  displays  allow  silent  and  pi  ivate 
communication.  This  is  relevant  since  almost  all  of  the 
communication  of  the  car  is  meant  for  the  driver  only.  Why 
bother  the  passengers  with  this  information,  especially  when 
they  are  working  or  communicating  with  others? 

d)  coded  information.  This  category  encompasses  all  types 
of  (abstractly)  coded  information  available  to  the  driver 
(e.g.,  speed,  engine  rpm,  and  fuel  supply).  Optimal  ways  of 
coding  must  be  determined  for  each  parameter.  Information 
of  this  type  could  be  presented  when  the  driver  asks  for  it,  or 
when  specific  conditions  or  limits  are  reached. 

e)  general.  Tactile  /  haptic  information  can  also  be  used  to 
guide  the  driver  to  different  locations  on  the  dashboard  or 
the  middle  console,  can  give  information  on  the  settings  of 
switches  and  buttons,  can  indicate  preference  points,  etc., 
without  the  need  of  the  visual  modality. 


2.  Integrating  tactile  displays  in  the  driver’s 


workplace 

For  optimal  safety,  the  workplace  allows  the  driver  to  have 
both  eyes  on  the  road,  both  hands  on  the  steering  wheel,  and 
both  feet  on  or  near  the  pedals.  This  safety  strategy  goes 
beyond  keeping  the  driver’s  attention  focussed  on  the  road.  A 
consequence  is  the  increasing  use  of  controls  on  the  steering 
wheel,  for  example  to  control  the  radio,  phone,  or  climate 
control  system.  However,  using  these  controls  as  display  is 
largely  unexplored.  Besides  the  primary  and  secondary 
controls,  tactile  displays  can  be  incorporated  in,  for  example, 
the  pedals,  the  seat  and  the  safetybelt. 

2.1.  Integrating  tactile  information  with  (existing) 
haptic  information 

Not  all  information  in  driving  is  visual.  For  example,  forces 
in  the  steering  wheel  provide  the  driver  with  indispensable 
information  on  vehicle  behaviour,  road  conditions,  etc. 
Therefore,  although  presenting  additional  haptic  information 
in  controls  may  be  an  attractive  option  to  present  information 
[10],  the  interference  with  naturally  available  haptic 
information  is  a  potential  disadvantage.  For  example, 
presenting  additional  forces  on  the  steering  wheel  to  display 
course  deviations  sensed  by  a  heading  control  system  might 
interfere  with  the  natural  information.  Using  the  tactile 
modality  -  such  as  a  small  vibration  presented  to  the  hands 
holding  the  steering  wheel  (not  on  the  steering  wheel  itself)  - 
may  not  have  these  consequences. 

2.2.  Joystick  control:  completely  integrating  haptic 
and  tactile  information 

The  introduction  on  a  large  scale  of  joystick  control  in 
automobiles  may  not  be  far  away.  Nowadays,  joystick 
control  is  already  applied  in  vehicles  for  the  handicapped 
driver.  The  major  advantage  of  the  use  of  a  joystick  instead 
of  a  steering  wheel  is  that  the  latter  is  a  major  source  of 
injuries  in  case  of  accidents.  Apart  from  the  design  issue  of 
providing  the  correct  force  feedback  information  in  joystick 
control,  it  also  opens  opportunities  to  integrate  haptic  and 
tactile  information.  Since  the  driver  has  his  or  her  hands  at  a 
fixed  position  on  the  joystick  (irrespective  of  the  manoeuvre) 
the  fingers  and  palm  are  interesting  locations  to  present 
tactile  information.  An  accompanying  advantage  is  that  the 
fingers  are  not  only  suited  as  display,  but  can  also  act.  This 
opens  up  the  possibility  to  design  a  fast,  intuitive 
stimulus-response  system. 

3.  Examples  of  applications:  SAFE:  Safety, 
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Assistance,  Fun,  and  Efficiency 

Below,  we  will  give  some  examples  of  tactile  display 
applications  in  the  car.  We  divide  the  applications  in  four 
categories  -  safety,  assistance,  fun,  and  efficiency  -  and 
provide  some  examples  of  each  category. 

Safety.  Tactile  information  can  help  to  make  driving  safer 
by  enabling  the  bands  on  controls’ strategy,  by  releasing  the 
visual  workload,  and  by  presenting  information  in  a  natural 
and  intuitive  way.  An  example  of  the  latter  is  displaying  cars 
from  behind,  cyclist  in  the  dead  angles  of  mirrors,  and 
crossing  pedestrians  in  the  periphery.  The  feasibility  study 
described  in  Section  4  provides  a  first  proof-of-concept  of 
an  in-car  tactile  navigation  display,  consisting  of  actuators  in 
the  seat. 

Assistance.  A  simple  and  straightforward  form  of  assistance 
is  applying  a  tactile  signal  on  the  appropriate  control.  For 
example,  a  small  tactile  vibration  on  the 
handbrake-release-button  can  indicate  the  moment  that  there 
is  enough  torque  on  the  wheels  to  pull  up  a  slope.  This 
probably  allows  even  the  most  inexperienced  driver  to  pull 
up  smoothly.  Van  Winsum  [11]  used  a  tactile  stimulus  on 
the  accelerator  pedal  to  indicate  when  a  speed  limit  was 
exceeded,  and  compared  this  way  of  displaying  with  an 
auditive  message.  The  tactile  display  resulted  in  a  decreased 
workload  and  faster  responses. 

Fun.  The  fun  of  tactile  information  probably  lies  in  the  fact 
that  it  enables  silent  and  private  communication  with  the 
machine,  which  is  to  a  large  extent  just  between  the  driver 
and  the  machine  and  of  no  concern  to  other  passengers.  An 
example  is  a  tactile  signal  on  the  door  handle  when  the  lights 
are  left  on.  The  latter  is  much  more  fun  than  an  annoying 
auditive  signal  that  everybody  in  the  parking  lot  can  hear. 

Efficiency.  Optimising  fuel  consumption  is  an  important 
issue  in  the  automobile  industry.  Tactile  displays  can 
support  a  more  efficient  gearshift  regime  by  the  direct 
coupling  of  display  and  actuator.  Indicating  the  moment  to 
shift  gears  can  be  realised  by  a  signal  on  the  appropriate 
hand  (e.g.,  via  the  steering  wheel).  The  optimum  will  be  the 
situation  in  which  the  fingers  are  already  on  the  gearshift 
controls  as  in  joystick  control  or  when  the  gearshift  controls 
are  integrated  in  the  steering  wheel. 

4.  Feasibility  study:  in-car  navigation  displays 

Research  has  shown  that  badly  designed  in-car  displays  can 
negatively  effect  safety.  For  example,  a  safety  evaluation 
[12]  of  in-car  displays  that  provide  real  time  traffic 


information  showed  a  50%  increase  of  the  number  of  critical 
traffic  situations  (as  judged  by  a  driving  instructor)  of  two 
badly  designed  interfaces  as  compared  to  two  well  designed 
interfaces.  Earlier  work  at  TNO  Human  Factors  has  shown 
that  a  tactile  display  that  presents  driving  behaviour 
feedback  results  in  a  lower  workload  than  an  auditive  display 
[11],  and  that  tactile  displays  can  be  used  effectively  for 
presenting  spatial  information,  such  as  the  direction  of 
waypoints  [6],  The  present  study  aims  to  investigate  the 
feasibility  of  an  in-car  tactile  navigation  display.  Special 
emphasis  will  be  on  workload.  The  hypothesis  is  that 
presenting  information  to  the  tactile  sense  will  decrease  the 
visual  burden  of  car  driving. 

4.1.  Design  of  the  visual  and  tactile  navigation 
symbols:  the  icons  and  vibrocons 

In-car  navigation  displays  present  relatively  simple  visual 
symbols:  Information  is  usually  only  presented  when  a 
course  change  is  required:  no  message  means  go  straight  on. 
The  information  consists  of  two  parts.  The  first  part  is  the 
direction  of  the  oncoming  (course)  change  (e.g.,  left,  right) 
and  the  second  is  the  remaining  distance  to  this  change.  The 
latter  may  be  dependent  on  the  road  category  or  driving 
speed.  We  adopted  this  simple  symbology  in  the  present 
experiment. 

Visual  and  tactile  symbols  were  made  for  three  distances: 
250,  150,  and  50  m.  Distance  was  indicated 

alphanumerically  for  the  icon,  and  by  timing  for  the 
vibrocon.  The  icons  (arrow  plus  distance)  were  presented  on 
a  separate  LCD  display  left  of  the  steering  wheel  with  a 
resolution  of  64  by  64  pix.  In  an  analogue  way,  we 
developed  the  tactile  coding  principles.  Based  on  a  pilot 
study,  the  following  were  the  most  appropriate  ways  to  code 
the  direction  and  distance,  respectively: 

Direction  of  the  course  change: 

$  location:  a  tap  on  the  left  or  right  side  of  the  body, 

$  motion:  simulate  motion  to  the  left  or  right  by 
activating  several  actuators  in  a  specific  spatio- 
temporal  pattern. 

Distance  to  the  next  waypoint: 

$  rhythm:  by  presenting  the  course  change  patterns  at 
closer  intervals,  a  smaller  distance  to  the  waypoint  is 
indicated, 

$  intensity:  a  higher  intensity  of  the  course  change 
pattern  indicates  a  smaller  distance. 

In  the  present  experiment,  we  used  rhythm  to  code  distance. 
The  timing  parameters  of  the  vibrocons  are  depicted  in  Table 
1.  Direction  was  coded  as  follows:  a  left  turn  was  indicated 
by  activating  four  tactors  under  the  left  leg  (vibrocon  left), 
turn  right  by  four  tactors  under  the  right  leg  (vibrocon  right), 
and  go  straight  by  the  consecutive  activation  of  the  tactors 
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from  back  to  front  under  both  legs  with  a  burst  duration  of 
60  ms  and  an  inter  stimulus  interval  of  0  ms  (vibrocon 
straight).  Table  1  reads  as  follows:  the  vibrocon  ’left  at  50 
m’  consists  of  activating  the  tactors  under  the  left  leg  five 
times  for  60  ms  with  inter  stimulus  intervals  of  52,  40,  32, 
and  20  ms. 


Table  1.  Timing  parameters  of  the  vibrocons. 


distance 

burst 

duration 

inter  stimulus 
interval 

number 
of  bursts 

250  m 

60  ms 

264,  264  ms 

3 

150  m 

60  ms 

264,212,  160, 

108,  52  ms 

6 

50  m 

60  ms 

52,  40,  32,  20  ms 

5 

4.2.  Method 


4.2.1.  Subjects,  task,  instruction,  and  performance 
measures.  Six  male  and  two  female  drivers  participated. 
Their  age  was  between  23  and  51  years.  All  were 
experienced  drivers,  had  normal  vision,  and  had  experience 
with  driving  simulators.  They  drove  12  experimental  routes 
in  a  driving  simulator,  preceded  by  a  familiarisation  run. 
During  this  run,  the  drivers  could  get  used  to  the  simulator, 
the  driving  environment,  the  icons  and  vibrocons,  and  the 
instructions.  The  instructions  were:  a)  to  employ  a  normal 
driving  style,  including  the  priority  rules  and  speed  limits 
applicable,  b)  to  follow  the  messages  of  the  navigation 
system,  c)  to  use  the  direction  sign  on  the  steering  wheel 
(i.e.,  left  /  right)  directly  after  receiving  the  navigation 
message,  and  d)  to  react  directly  on  the  stimuli  of  the 
Peripheral  Detection  Task  (PDT,  see  below)  by  pressing  a 
finger  switch. 

During  the  complete  route,  a  PDT  was  presented.  The 
PDT  is  based  on  the  narrowing  of  the  visual  view  as 
function  of  workload  [13,  14,  15].  The  PDT  is  able  to 
measure  the  workload  during  driving  and  to  measure 
workload  peaks  of  short  duration.  [11].  The  PDT  presented 
red  squares  in  the  left  periphery  of  the  visual  field  (between 
11-23°,  2-4°  above  the  horizon)  for  Is.  The  stimuli  were 
presented  with  a  random  interval  between  3-5  s.  The 
participant’s  task  was  to  react  as  fast  as  possible  to  the 
stimulus  by  pressing  a  finger  switch,  attached  to  the  index 
finger  of  the  right  hand. 

The  12  runs  were  divided  over  three  conditions  based  on 
the  modality  in  which  the  navigation  information  was 
presented:  visual  only,  tactile  only,  and  visual  plus  tactile. 
The  runs  were  blocked  accordingly.  After  each  block,  the 
participant  filled  in  the  BSMI,  a  standardised  subjective 
mental  effort  rating  [16],  The  BSMI  has  an  interval  scale. 


Three  other  perfomance  measures  were  calculated  for  each 
route:  the  mean  reaction  time  to  the  PDT,  percentage 
detected  PDT  stimuli,  and  the  reaction  to  the  navigation 
information,  expressed  in  the  remaining  distance  to  the  next 
waypoint  at  the  moment  of  indicating  direction.  The  order  of 
the  conditions  was  balanced  over  the  subjects,  the  different 
routes  were  balanced  over  the  conditions. 

4.2.2.  Apparatus.  The  experiment  was  run  in  the  TNO 
LoCo  simulator.  This  driving  simulator  has  a  180  x  30°  field 
of  view,  rear-view  mirrors,  haptic  feedback  in  the  controls, 
automatic  gear  change,  and  interactive  traffic.  The  database 
chosen  for  the  present  experiment  was  an  urbanised  area, 
measuring  3x3  km.  Included  were  town  roads,  traffic  signs, 
interactive  traffic  lights,  interactive  traffic,  intersections  with 
different  priority  situations,  and  roundabouts.  Primary  data 
(including  position,  heading,  steering  wheel  angle,  etc.)  were 
stored  with  60  Hz  sampling  frequency.  The  tactile  actuators 
were  Special  Instrument  MiniVib  4  vibrators.  These  are 
small  boxes  (18  x  22  mm)  and  present  a  250  Hz  vibration. 
The  vibrators  were  mounted  in  the  seat  (four  for  each  thigh, 
in  a  straight  line  from  rear  to  front)  with  a  centre-to-centre 
distance  of  4  cm).  Based  on  the  distance  to  the  next  waypoint 
(250,  150,  50  m),  the  navigation  symbol  (icon,  vibrocon,  or 
both)  was  presented.  Each  icon  was  presented  till  the  next 
icon  appeared  or  the  wapoint  was  reached. 

4.3.  Results  and  discussion 

The  results  of  each  performance  measure  were  analysed  by  a 
within  subjects  analyses  of  variance  with  modality  (3)  as 
independent  variable.  Significant  effects  (p  <  .05)  were 
present  for  the  BSMI  (mental  effort)  rating,  and  the  distance 
to  the  waypoint  at  which  the  subject  reacted  to  the  navigation 
message.  The  means  are  presented  in  Figure  1  and  2, 
respectively. 
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Figure  1.  The  effect  of  modality  on  the  mental 
effort. 
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Figure  2.  Effect  of  modality  on  the  remaining 
distance  to  the  oncomming  waypoint  when  the 
driver  reacts  to  the  navigation  message. 

The  analysis  of  the  mental  effort  shows  that  the  rating  in 
the  visual  only  condition  is  about  25%  higher  than  in  the 
tactile  only  condition.  The  rating  in  the  visual  +  tactile 
condition  is  in  between.  The  same  order  of  conditions  is  also 
present  on  the  remaining  distance  to  the  waypoint  at  which 
the  driver  reacts  to  the  navigation  message  (here  a  higher 
score  indicates  better  performance).  The  means  indicate  that 
drivers  react  faster  to  the  tactile  messages  than  to  the  visual 
messages,  while  the  score  for  the  tactile  +  visual  condition  is 
close  to  the  tactile  only  score.  Since  the  first  navigation 
message  is  presented  250  m  before  the  waypoint,  it  means 
that  the  mean  distance  to  react  to  the  message  is  80  m  in  the 
tactile  condition  and  92  m  in  the  visual  condition,  an 
increase  of  15%. 

Both  effects  show  the  superiority  of  the  tactile  modality 
over  the  visual  modality  for  presenting  navigation  messages: 
Drivers  react  faster  to  the  message,  and  with  less  mental 
effort.  The  fact  that  adding  the  visual  modality  to  the  tactile 
messages  reduces  performance  may  be  due  to  the  fact  that 
some  drivers  check  the  tactile  message  by  looking  at  the 
visual  display,  especially  during  the  first  runs  with  the  tactile 
displays.  Afterwards  they  stated  that  it  takes  some  time  to 
learn  to  ’trust’  the  unusual  tactile  messages.  This  extra  check 
reduces  performance  and  increases  the  mental  effort. 

The  MANOVA  on  the  performance  in  the  PDT  did  not 
result  in  a  highly  significant  of  condition  (p  <  0.10).  The 
means  for  the  percentage  detected  stimuli  and  the  reaction 
time  are  given  in  Figure  3.  Both  PDT  measures  showed 
better  performance  in  the  tactile-only  condition. 

The  PDT  data  objectively  confirm  the  result  of  the 
mental  effort  rating  scale:  the  tactile  navigation  display 
results  in  a  decreased  workload  as  compared  to  a  visual 
navigation  display. 


visual  tactile  visual  +  tactile 

modality 


Figure  3.  Effects  of  modality  on  the  performance  in 
the  peripheral  detection  task. 

4.3.  Conclusions 

The  results  of  the  feasibility  study  show  the  superiority  of  the 
tactile  navigation  display  over  the  visual  display  in  the 
current  set-up:  faster  reaction,  lower  mental  effort  and  lower 
workload.  These  are  the  first  indications  that  using  the  tactile 
modality  in  automobiles  may  improve  the  quality  and  safety 
of  the  man-machine-interface.  However,  we  recommend 
performing  field  experiments  with  an  instrumented  vehicle 
and  expert  opinions  to  measure  the  effects  on  vehicle  control 
and  traffic  behaviour 

.  However,  it  may  be  expected  that  the  results  in  real 
driving  may  be  even  more  pronounced.  There  are  several 
reasons  for  this.  For  example,  in  the  present  set-up,  the  PDT 
stimuli  were  presented  on  the  same  side  (left)  as  the  visual 
navigation  messages.  With  the  visual  display  located  on  the 
opposite  side,  effects  of  a  narrowing  field  of  view  will  be 
stronger  for  the  visual-only  condition.  Another  important 
aspect  is  that  in  the  driving  simulator  the  navigation  display 
and  the  outer  scene  are  presented  at  the  same  distance.  On 
the  road,  the  driver  must  adjust  to  the  light  levels  and  focal 
distance  when  switching  between  the  navigation  display  and 
the  outer  scene.  Furthermore,  some  of  the  participants 
indicated  that  it  takes  more  (training)  time  for  them  to  learn 
to  ’trust’  the  tactile  messages  than  one  three-minute  training 
run. 

5.  General  Conclusions 

Using  vibro-tactile  displays  in  cars  appears  to  be  a  useful 
concept,  as  the  list  of  potential  applications  and  initial 
proof-of-concept  study  shows.  Its  major  promise  is  to 
improve  the  safety,  efficiency,  and  fun  of  car  driving.  The 
technological  feasibility  is  certainly  high;  the  required 
displays  can  be  constructed  from  cheap  and  robust  elements 
that  are  now  on  the  market.  Former  research  showed  the 
superiority  of  a  tactile  display  over  an  auditive  display  for 
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warning  the  driver  when  exceeding  a  speed  limit.  1  he 
present  experimental  data  show  that  a  tactile  navigation 
display  results  in  faster  reactions  to  the  navigation  messages 
and  lower  workload  and  mental  effort  than  a  visual  display. 
Vibro-tactile  displays  seem  to  offer  a  valuable  answer  to  the 
demand  of  the  automotive  industry  to  improve  the 
man-machine  interfaces  in  cars. 
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Abstract 

This  paper  presents  the  necessity  and  the  concept  of  a 
new  approach  to  control-units  for  technical  devices  with 
high  functionality. 

Due  to  the  increasing  contact  of  humans  with  technical 
devices  in  everyday  life  and  the  growing  complexity  of 
these  devices,  the  man-machine  communication  gains  in 
importance.  The  state  of  the  art  man-machine  interfaces 
to  operate  complex  technical  devices  are  conventional 
push-buttons  and  touch-screens.  Because  both  lack  in 
either  flexibility  and  clarity,  or  in  haptic  feedback,  inves¬ 
tigations  about  new  solutions  are  necessary. 

The  approach  presented  is  characterized  by  a  flat  con¬ 
trol-panel,  where  buttons  can  arise  in  variable  configura¬ 
tions.  The  buttons  are  programmable  in  size  and  haptic 
properties.  The  panel  will  comprise  of  single  segments, 
each  of  them  consisting  of  an  actuator  and  a  position 
sensor,  which  are  currently  in  development. 

The  requirements,  possibilities,  and  difficulties  of  this 
approach  are  discussed. 

1.  Introduction 

The  communication  between  man  and  machine  nowa¬ 
days  gains  in  importance.  On  the  one  hand  people  are 
more  often  in  contact  with  technical  systems  in  everyday 
life.  Transportation,  communication,  professional  and 
leisure-time  activities  are  almost  unthinkable  without 
using  technical  devices.  On  the  other  hand  these  devices 
keep  on  increasing  their  number  of  features.  The  success 
of  modern  microelectronics  enables  the  implementation  of 
additional  functionality  with  almost  no  additional  costs 
[1],  In  summary  there  is  a  trend  towards  technical  devices 
becoming  more  powerful  and  complex. 

2.  Problem 

Unfortunately  the  increase  of  functionality  does  not 
imply  only  advantages  for  the  user,  especially  with  regard 
to  the  man-machine  interface.  The  additional  features 
have  to  be  selected  and  controlled  by  the  user,  what  often 


changes  the  usability  of  the  device  for  the  worse.  People 
suffer  from  “feature  shock”,  meaning  they  are  unable  to 
use  the  functionality  due  to  the  difficulty  in  operating  the 
machine.  This  is  a  serious  problem,  as  the  features  of 
devices  from  competing  companies  grow  closer  together, 
and  therefore  the  man-machine  interface  becomes  deci¬ 
sive  for  sales  [2], 

2.1  Conventional  solution 

There  are  two  common  ways  to  handle  the  high  func¬ 
tionality  of  technical  devices.  One  possibility  is  to  provide 
a  huge  control  panel  with  a  lot  of  buttons  and  switches. 
Besides  the  drawback  of  a  confusing  appearance  of  such  a 
panel,  the  available  space  for  the  control  unit  limits  this 
solution.  In  order  to  be  used  by  human  fingers,  every 
single  control  element,  like  a  push-button,  must  not  fall 
below  a  certain  size. 

Another  more  promising  way  to  handle  the  large 
amount  of  features  is  with  the  use  of  touch-screens.  The 
touch  sensitive  displays  offer  the  opportunity  of  a  menu- 
driven  handling,  similar  to  what  is  used  in  computer  soft¬ 
ware.  The  menu  structure  is  based  upon  the  idea  of  not 
offering  all  selectable  functions  at  once,  but  to  divide 
them  into  different  levels.  Therefore  a  lot  of  functions  can 
be  offered  by  displaying  only  a  manageable  number  of 
control  elements.  Today  touch-screens  are  already  often 
used  for  pda’s,  ticket-machines,  cash-dispensers  etc. 

The  major  disadvantage  of  touch-screens  is  the  missing 
haptic- feedback,  that  ordinary  push-buttons  provide  in 
contrast  to  the  virtual  push-buttons  of  touch-screens.  This 
results  in  an  unexpected  perception  during  the  operation. 
People  are  used  to  feel  the  contact  between  the  finger  and 
the  control  element,  the  force  exerted  by  the  button  to  the 
finger,  the  movement  during  the  depressing,  and  the  hap¬ 
tic  “click”  when  the  electric  contact  is  closed.  These  sen¬ 
sations  form  the  basis  for  using  control  elements  in  an 
intuitive  and  automatic  way  without  the  need  of  attention 
[3].  As  the  user  only  pushes  on  a  rigid  screen  made  of 
glass  while  using  a  touch-screen,  the  operation  does  not 
happen  intuitively.  Figure  1  depicts  the  differences  of  the 
information  flow  between  using  a  conventional  push¬ 
button  and  a  touch-screen. 
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(a) 


relevant  Feedback  irrelevant  Feedback 

Figure  1.  Comparison  of  the  sensory  feedback  while  using  (a) 
a  conventional  push-button  and  (b)  a  touch-screen.  The  position 
and  exerted  force  (mechanical  impedance)  of  the  push-button 
change  during  depressing,  which  is  sensed  by  the  haptic  chan¬ 
nel.  The  rigid  touch-screen  provides  only  a  haptic  sensation  of 
the  contact  between  finger  and  screen.  An  additional  feedback  to 
another  sensory  organ  has  to  be  generated  artificially  to  confirm 
the  operation  of  the  virtual  button  on  the  screen. 

Even  if  people  are  able  to  learn  new  patterns  of  behav¬ 
iour  for  controlling  machines  by  a  lot  of  training,  it  is 
questionable  if  the  human  should  be  forced  to  do  so,  es¬ 
pecially  if  elderly  people  have  to  use  these  machines  (e.g. 
ticket-machine)  too.  We  think  that  new  technology  should 
adapt  to  the  human,  not  vice  versa.  This  view  is  also 
stated  by  other  authors  [4]. 

3.  New  approach 

As  haptic  feedback  is  expected  by  the  user  and  helpful 
for  intuitive  machine  operation,  it  appears  reasonable  to 
investigate  the  haptic  aspect  in  man-machine  communica¬ 
tion.  The  claim  for  haptic  feedback  during  the  operation 
of  machines  in  order  to  prevent  uncertainty  and  enable 
fast  and  comfortable  controlling  is  mentioned  by  several 
authors  [5][6j|7|.  Therefore  we  are  working  on  a  new 
approach  to  operate  the  enormous  functionality  of  modern 
technical  devices  without  neglect  of  the  haptic  require¬ 
ments.  The  solution  seems  to  be  a  combination  of  both  the 
flexibility  of  touch-screens  and  the  haptic  stimuli  of  ordi¬ 
nary  push-buttons. 

We  envisage  a  flexible  control-panel  that  is  able  to  of¬ 
fer  different  numbers  of  buttons,  which  are  -  in  contrast 
to  the  virtual  buttons  on  touch-screens  -  indeed  to  be 
pressed  down  by  the  user  similar  to  conventional  push¬ 
buttons.  Depending  on  the  current  situation  or  level  of  the 
menu-structure,  the  control  pane!  is  able  to  generate  the 


required  number  of  buttons  with  programmable  shape  and 
size. 


3.1  Concept  and  applications 

The  concept  of  a  flexible  control-panel  with  haptic 
feedback  can  be  realized  as  follows:  Imagine  a  flat  area 
consisting  of  many  single  segments,  which  can  arise  in¬ 
dependently  from  each  other  as  illustrated  in  figure  2.  In 
this  way  different  numbers  of  buttons  with  variable  forms 
and  sizes  can  be  generated.  With  a  specific  controlling  the 
buttons  can  put  up  a  resistance  when  pressed  down  by  the 
user.  Thus  a  menu-driven  handling  with  adequate  haptic 
feedback  is  possible. 


Figure  2.  Sketch  of  a  control-panel  with  buttons  variable  in 
number  and  size.  The  panel  comprises  of  single  segments. 

The  range  of  control-panels  designed  in  this  way  is 
very  wide.  In  many  cases  they  could  substitute  touch¬ 
screens.  But  even  new  applications  can  be  imagined.  A 
universal  remote  control  for  TV-set,  video-recorder,  satel¬ 
lite-receiver  and  so  on  could  be  designed,  that  offers  for 
every  device  just  the  accessory  control-elements.  Such  a 
remote  control  differs  from  other  universal  controls  on  the 
market  in  its  clear  arrangement  of  the  buttons. 

Moreover  the  force  exerted  by  the  buttons  of  the  de¬ 
scribed  control-panel  is  variable,  as  the  conditions  of  the 
segments  are  programmable.  So  for  instance  an  user  spe¬ 
cific  adaptation  is  conceivable.  Depending  on  the  user’s 
desire,  more  or  less  force  is  needed  to  press  the  buttons 
down.  Another  possibility  is  to  adapt  the  characteristic  of 
a  button  to  its  function.  For  example  the  more  important 
or  serious  the  launch  of  a  function  is,  the  more  force  is 
needed  to  push  the  referring  button  down.  Additionally 
some  more  information  can  be  transferred  to  the  user  with 
a  special  behaviour  like  vibrations  of  the  buttons.  An 
event  like  a  malfunction  can  be  indicated  in  this  way. 


4.  State  of  the  project  and  further  tasks 

In  older  to  demonstrate  the  advantages  and  perform¬ 
ance  of  the  concept  presented,  a  control-panel  for  testing 
purposes  should  be  developed. 

Currently  the  design  of  the  segments,  which  will  fi¬ 
nally  form  the  control-panel,  is  in  progress.  As  the  posi¬ 
tion  and  the  exerted  force  of  the  segments  have  to  be 
controlled,  every  single  segment  is  a  system  consisting  of 
an  actuator  and  a  sensor. 

In  order  to  ensure  a  wide  range  of  different  program¬ 
mable  combinations  of  buttons  with  various  shapes,  the 
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dimensions  of  the  segments  should  be  relatively  small. 
For  the  demonstration  system  the  mechanical  design 
should  not  exceed  a  maximum  size  of  lxl  cm2  per  seg¬ 
ment.  Besides  the  limited  space,  the  actuator  needs  to 
show  some  more  important  performance  characteristics. 
A  minimum  pressure  of  20  mN/mm2  is  required  to  create 
a  unique  haptic  impression  during  the  depressing.  So  a 
button  of  the  size  of  1  cm2  can  exert  a  force  of  2  N,  which 
is  an  average  value  of  conventional  push-buttons. 

Several  authors  discuss  the  force  output  resolution  for 
force  feedback  devices.  According  to  SRINIVASAN  [8]  a 
resolution  of  0,01  N  is  necessary  to  make  full  use  of  hu¬ 
man  haptic  capabilities,  whereas  Shimoga  [9]  recom¬ 
mends  0,05  N,  respectively  0,02  N/cm2. 

The  stroke  of  the  actuator  has  to  be  at  least  5  mm.  This 
is  not  so  much  necessary  for  the  haptic  impression,  as  to 
ensure  a  visible  difference  between  an  inactive  and  an 
active,  which  means  extended,  segment. 

The  choice  of  possible  actuators  is  strongly  limited  by 
these  constraints.  On  closer  examination  only  an  electro¬ 
dynamic  actuator  seems  to  fulfil  the  demands.  Moreover 
such  an  actuator  has  the  advantage  of  a  linear  relation 
between  the  active  current  and  the  exerted  force.  This 
enables  a  system  without  force  sensors,  as  the  force  can 
be  calculated  if  the  current  is  known. 

As  the  realised  control-panel  should  provide  buttons 
with  an  appropriate  haptic  feedback,  the  exerted  force 
must  not  be  constant  during  the  depressing.  There  is  the 
need  for  a  rising  resistance  until  the  button  switches, 
afterwards  a  decrease.  This  characteristic  behaviour  is 
called  “haptic  click”  and  supports  an  intuitive  operation 
[5].  In  order  to  provide  such  behaviour,  the  position  of  the 
segments  need  to  be  measured  by  a  sensor. 

Similar  to  the  design  of  the  actuator,  the  sensor  design 
has  to  be  adapted  to  the  human  haptic  capabilities.  Al¬ 
though  the  threshold  to  detect  static  height  differences  by 
skin  deformation  at  the  human  fingertip  is  10  pm  [10], 
Rosenberg  advises  a  position  resolution  of  25  pm  [11] 
for  a  high  quality  force  feedback  system.  Further  the  size 
of  the  sensor  should  not  exceed  the  dimensions  of  the 
actuator.  With  respect  to  cost  efficiency,  an  optical  sen¬ 
sor,  which  evaluates  reflecting  light  of  an  infrared  light- 
emitting  diode,  will  be  used  for  the  first  conception. 

After  completing  the  design  of  the  actuator  and  the 
sensor,  a  prototype  of  a  single  segment  has  to  be  fabri¬ 
cated.  An  evaluation  referring  to  the  mechanical  and  elec¬ 
trical  behaviour  will  show  whether  a  redesign  is  neces¬ 
sary.  If  the  resolution  of  the  exerted  force  and  of  the 
measured  position  matches  the  expectations,  the  next  step 
is  to  compose  an  array  consisting  of  several  segments. 
The  mechanical  design  of  this  array  requires  much  devel¬ 
opment  effort,  as  a  modular  concept  is  preferred  to  enable 
a  simple  assembly  and  serviceable  operation. 

Beside  the  mechanic  and  electronic  development  of  the 
actuator-sensor-system,  investigations  about  the  control¬ 
ling  have  to  be  done.  Depending  on  the  desired  appear¬ 


ance  of  the  new  created  control-panel,  the  array  has  to  be 
configured  in  such  a  way,  that  the  correct  number  of  but¬ 
tons  arises  with  the  expected  size  and  location.  Therefore 
the  stroke  and  exerted  force,  of  every  single  segment  has 
to  be  controlled  independently  from  other  segments. 

A  major  challenge  is  to  provide  a  uniform  movement 
of  all  segments  forming  one  of  those  buttons,  even  if  not 
all  segments  arc  touched  and  therefore  depressed  by  the 
user’s  finger.  Otherwise  the  button  would  not  visually 
appear  as  a  self-contained  unit.  This  would  confuse  the 
user. 

Finally  the  prototype  of  the  new  approach  to  control- 
units  has  to  be  evaluated  by  thorough  investigations.  Both 
the  technical  performance  and  the  usability  for  everyday 
applications  have  to  be  checked. 
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Abstract 

In  order  to  create  an  algorithm,  which  predicts  the  endpoint  of  a  cursor  movement,  an  experiment  was  performed  to 
reveal  characteristics  of  cursor  movements.  In  the  experiment  subjects  had  to  click  on  a  button  in  the  center  of  the 
screen  after  which  they  moved  the  cursor  onto  a  target  which  subsequently  appeared.  After  the  target  was  reached  it 
disappeared.  Targets  were  radially  located  at  angles  of  n*  45  degrees  (0  <n<7),  and  at  distances  of  150,  250  or  350 
pixels  (34  pixels/cm).  Each  of  nine  subjects  executed  this  experiment  three  times,  using  either  a  mechanical  mouse,  an 
optical  trackball  or  a  mechanical  force  feedback  trackball.  The  variables  included  in  the  analyses  were  curvature, 
variability  around  the  mean  path  and  the  distance  whereon  peak  velocity  occurred.  The  results  show  that  there  is  no 
need  to  compensate  for  a  curvature  other  than  zero  in  the  algorithm.  As  the  devices  differ  in  their  amount  of  variability 
and  the  location  of  the  peak  velocity  one  algorithm  for  each  device  is  preferable.  Within  one  device  it  could  be 
considered  to  have  a  less  accurate  prediction  or  to  make  a  special-case  algorithm  for  direction  and  distance.  To 
increase  the  accuracy  even  more  one  could  choose  to  make  an  algorithm  for  each  person  separately. 


1.  Introduction 

Technical  development  and  cognitive  ergonomical 
research  on  human-computer  interaction  have  resulted 
in  more  userfriendly  interfaces.  More  and  qualitatively 
better  colors  and  sounds  are  being  used  in  a  more 
efficient  and  attractive  way. 

Another  phenomenon  starting  to  integrate  in 
human-computer  interaction  is  the  use  of  the  sense  of 
touch.  This  can  be  done  by  tactual  or  force  feedback 
devices.  For  instance,  playing  games  with  a  force 
feedback  joystick  is  already  very  common.  Other  areas 
wherein  tactual  feedback  is  developing  rapidly  are 
medical  training  (e.g.  the  PHANToM)  and  blind 
computer  users  (e.g.  the  Virtual  Reality  mouse). 

In  every-day  computer  use  the  use  of  tactual 
feedback  is  not  yet  very  common.  However,  tactual 


feedback  can  be  very  helpful  in  this  area  as  well. 
Keyson  [4J  showed  that  target  acquisition  with  force 
feedback  was  10-15%  faster  than  without. 
Unfortunately,  force  feedback  in  this  area  clearly  needs 
restriction,  because  getting  force  feedback  every  time 
the  cursor  crosses  an  object  does  not  make  any  sense, 
in  fact  this  will  be  very  disturbing.  We  think  that  the 
advantages  of  force  feedback  in  this  area  are  best  used 
when  it  helps  the  user  toward  an  object  he/she  wants  to 
reach  while  not  being  disturbed  by  other  objects. 
Therefore  the  computer  needs  to  know  what  the  users’ 
target  is  in  advance. 

For  this  purpose  we  want  to  create  an  algorithm  that 
can  predict  the  users'  intended  target  using  the  initial 
part  of  a  cursor  movement.  Tactual  feedback  can  then 
be  released  to  help  the  user  moving  toward  the  target 
and  away  from  non-targets.  In  order  to  create  such  an 
algorithm  we  need  knowledge  about  the  characteristics 
of  cursor  movements  toward  a  target.  Characteristics 
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can  he  the  curvature  of  paths,  the  variability  of  paths 
around  the  mean  path  and  the  velocity  profiles.  When 
we  know  these  path’s  statistics  it  is  possible  to  use  the 
initially  observed  direction  and  velocity  of  a  cursor 
movement  to  predict  the  location  ol  the  users  intended 
target.  The  amount  of  variability  can  he  used  to 
estimate  the  maximum  tat  get  resolution  at  which  an 
algorithm  can  predict  the  user’s  target  with  a  certain 
pre -specified  probability.  However,  these 
characteristics  could  be  influenced  by  factors  like 
direction,  distance,  device  and  user. 

In  this  paper  we  describe  an  experiment  in  which 
we  explore  cursor  movements  from  a  fixed  starting 
point  to  targets  located  at  three  diltcrcut  distances  and 
in  eight  different  directions.  When  the  factors 
mentioned  do  not  influence  the  characteristics  it  is 
possible  to  create  a  general  prediction -algorithm  which 
is  valid  for  all  eases.  When  the  factors  do  influence  the 
characteristics,  it  might  be  useful  to  create  special- ease 
algorithms  or  to  give  in  to  the  accuracy  of  the 
prediction. 


2.  The  experiment 

This  experiment  aimed  to  analyse  cursor  trajectories 
created  by  computer  users  in  eight  different  directions, 
three  different  distances  and  three  different  devices. 
The  data  sampled  were  the  co-ordinates  and  time. 
From  these  data  we  calculated  mean  paths  curvatures, 
the  variability  around  the  mean  paths  and  velocity 
profiles. 

In  the  analyses  the  following  aspects  were  involved: 

-  Per  device  we  tested  whether  the  mean  path 
curvatures  in  each  direction  and  on  each  distance 
differed  from  zero.  A  non-zero  curvature  would 
mean  that  (lie  trajectories  arc  not  straight,  but 
curved  in  a  certain  way.  When  this  is  the  case  it 
could  be  useful  to  compensate  for  it  in  the  general 
prediction-algorithm.  Also,  when  this  is  the  case 
for  some  of  the  mean  paths,  for  example  in  one 
direction,  it  could  be  useful  to  alter  the  general 
algorithm  for  this  occasion. 

We  investigated  the  influence  of  the  different 
factors  on  the  variability  around  the  mean  path. 
The  amount  of  variability  defines  the  accuracy 
with  which  the  prediction  can  he  made.  More 
variability  means  a  less  accurate  prediction.  For 
example  when  90%  of  the  paths  vary  within  20 
pixels  around  the  mean  path,  an  accuracy  of  90% 
can  be  readied  when  objects  are  located  40  pixels 
away  from  each  other.  That  is  two  times  the 
variation  around  each  mean  path.  By  decreasing 
the  resolution  of  the  objects  the  accuracy  of  the 
prediction  will  increase. 


-  We  calculated  at  what  fraction  of  the  target 
distance  velocity  was  maximum.  In  the  general 
algorithm  this  fraction  sets  the  multiplication 
factor  to  assess  the  target  distance.  As  well  as  the 
other  variables,  this  fraction  could  be  influenced 
by  direction,  distance,  devices  or  user  and 
therefore  needs  special-case  multiplication 
factors. 

2.1.  Material  and  techniques 

The  experiment  was  done  with  three  devices;  a 
mechanical  mouse,  an  optical  trackball  and  a 
mechanical  force  feedback  trackball.  The  mechanical 
mouse  we  used  was  a  Logitec  mouse.  We  call  this  a 
mechanical  mouse  because  the  movement  of  the  cursor 
is  related  to  the  mechanical  movement  of  the  ball 
within  the  mouse.  The  optical  trackball  was  the 
Logitec  Marble  Mouse.  The  force  feedback  trackball  is 
a  trackball  with  tv/o  motors,  which  can  roll  the  ball  in 
either  x-  or  y-direction  (2].  Without  activating  the 
motors  this  trackball  acts  as  a  normal  trackball  except 
that  it  is  rather  big  (diameter  55mm)  and  that  the 
friction  is  greater  as  a  result  of  the  contact  wheels  of 
the  motors. 


Figure  1.  The  mechanical  force  feedback 
trackball  (opened) 

We  used  a  PC  (Pentium  11)  and  a  17-incli  Dell-monitor 
with  1248*768  pixels  for  the  sessions  with  the 
mechanical  mouse  and  die  optical  trackball.  For  the 
session  with  the  force  feedback  trackball  we  used  a 
386-PC  and  a  17-inch  Philips  Monitor.  On  the  screen 
an  interface  was  created  with  Visual  Basic  5.0.  The 
target  consisted  of  a  black  circle  30  pixels  in  diameter, 
radially  located  at  angles  of  n  *  45  degrees  (0  <  n  <  7), 
and  at  distances  of  150,  250  or  350  pixels  (34 
pixels/cm). 
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During  each  trial  x-  and  y-co-ordinates  (in  pixels)  and 
the  system  time  were  sampled.  This  was  done  at  50  Hz. 

2.2.  Subjects 

Subjects  were  nine  colleagues,  five  men  and  four 
women.  The  mean  age  was  29.6  with  a  standard 
deviation  of  4.5.  All  subjects  were  experienced  mouse 
users,  but  had  no  experience  with  trackballs. 

2.3.  Procedure 

In  the  experiment  subjects  had  to  click  on  a  button  in 
the  center  of  the  screen.  A  target  appeared  in  one  of 
eight  directions  and  at  one  of  three  distances  to  which 
the  subjects  moved  the  cursor.  After  the  target  was 
reached  it  disappeared.  The  next  trial  started  when  the 
button  was  clicked. 

2.4.  Design 

The  experiment  was  divided  into  three  sessions.  In 
each  session  another  device  was  used,  the  order  of 
which  was  random. 

Each  session  consisted  of  a  practice  and  an 
experimental  part.  The  practice  was  included  to  get  the 
subject  accustomed  to  the  device  and  the  task.  In  the 
practice  part  each  target  was  presented  two  times  in  a 
random  order.  In  the  experimental  part  each  target  was 
presented  ten  times,  also  randomly.  A  session  lasted 
for  about  15  minutes. 

2.5.  Variables 

Before  calculating  the  curvature  and  the  variability  we 
performed  a  resampling  on  the  data,  because  the 
original  sampling  was  time  controlled.  This  means  that 
velocity  of  the  movement  influences  the  spacing 
between  the  samples.  To  calculate  the  curvature  of  a 
trajectory,  however,  we  want  equal  spatial  distances 
between  the  samples,  because  then  every  part  of  the 
trajectory  contributes  equally  to  the  calculation  of  the 
curvature.  Therefore,  for  each  trajectory  new  samples 
were  calculated,  by  interpolation,  for  every  one-percent 
of  the  total  travelled  distance  (distance  of  the  trajectory 

itself)' 

To  assess  the  curvature,  the  paths  were  first  rotated 
until  start  and  end  positions  were  on  the  positive  x- 
axis.  Then  the  distance  between  each  calculated  co¬ 
ordinate  and  the  x-axis  was  taken.  This  distance  is 
equal  to  the  accompanying  y-co-ordinate,  both  with 
respect  to  value,  sign  and  unit.  Curvature  was  then 
defined  as  the  mean  value  of  this  distance  along  the 
path  [1].  E.g.,  an  average  distance  of  +3  means  a 
positive  curve  of  three  pixels. 


The  variability  of  the  paths  is  represented  by  the 
standard  deviation  of  all  sample  points  around  the 
mean  path  of  the  ten  trajectories  performed  per  subject 
per  direction  and  per  device.  The  standard  deviation  is 
calculated  over  each  ten-percent  section  of  the  whole 
path. 


3.  Results 

Figure  2  shows  the  mean  paths  and  the  variability 
around  it  for  each  device  and  each  of  the  three  different 
distances.  Standard  deviation  bars  were  computed  and 
plotted  at  every  one-tenth  path  length.  From  this  figure 
a  number  of  statistics  can  be  derived  for  both  curvature 
and  variability. 

3.1.  Curvature 

The  mean  paths  shown  in  figure  2  look  rather  straight, 
but  this  can  not  be  simply  assumed  to  be  the  case  (as 
Murata  [5]  did).  We  tested  the  calculated  curvature 
against  zero  for  every  direction  separately.  It  was  also 
tested  separately  per  device  in  order  to  observe 
possible  systematic  effects  of  the  mechanics  of  each 
device. 

Curvature  is  significantly  different  from  zero  on 
almost  every  direction  for  almost  every  device.  Paths 
made  with  the  mechanical  force  feedback  trackball  are 
three  to  four  times  more  curved  than  paths  with  the 
other  two  devices.  For  the  mechanical  mouse  and  the 
optical  trackball  curvatures  range  from  -2  till  +  6.  The 
optical  trackball  shows  curvature  from  -8  to  +11. 
However,  including  a  curvature  variable  into  the 
prediction  algorithm  will  only  be  effective  when  the 
curvature  is  more  than  the  amount  of  variability. 
Looking  at  Figure  2,  this  will  not  be  the  case. 

3.2.  Variability 

An  overall  mean  variability  of  12.63  pixels  was 
calculated.  A  repeated  measures  ANOVA  on 
variability,  with  device,  distance  and  direction  as 
within-subjects  factors,  showed  that  there  is  a 
significant  influence  from  device  (F2, 7=25.526,  pc.01), 
distance  (F2,7=27.866,  pc.OOl)  and  direction 
(F7, 2=60.770,  p<.05)  as  well  as  an  interaction  effect 
between  device  and  distance  (F4j5=7.434,  p<.05).  Also 
a  significant  subjects  effect  was  found  (Fu=296.05, 
pc.OOl).  Post  hoc  tests  showed  that  the  variability  of 
the  paths  depends  on  the  first  three  factors  in  the 
following  way: 

(1)  direction:  for  each  device  the  variability  in  opposite 
directions  equal  each  other.  Also,  movements  with 
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each  device  are  significantly  less  variable  in  horizontal  (3)  control  device:  the  overall  variability  is  highest 

direction  than  movements  in  the  oblique  directions.  with  the  optical  trackball  (16.3  pixels)  and  lowest  with 

Vertical  movements  are  somewhat  more  variable  than  the  mechanical  mouse  (9.7  pixels).  These  results  are 

horizontal  movements,  but  this  effect  is  not  significant.  comparable  with  results  found  in  an  earlier  experiment 

(2)  distance:  the  nearest  target  results  in  the  least  by  Keuning  and  Houtsma  [3]. 

variability  (10,6  pixels),  except  for  the  mechanical 
force  feedback  trackball  which  yields  a  more  complex 
pattern. 


Figure  2:  Mean  paths  and  variability  for  A.  mechanical  mouse,  B.  optical  trackball,  C.  mechanical 
force  feedback  trackball,  separately  for  each  distance  a.  150  pixels,  b.  250  pixels,  c.  350  pixels. 
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3.3.  Velocity  peak 

Averaged  over  all  directions,  distances,  control  devices 
and  subjects  the  peak  velocity  was  reached  at  53%  of 
the  total  distance.  A  repeated  measures  ANOVA  on 
the  location  of  peak  velocity  showed  that  there  is  a 
significant  influence  of  device  (F27=5.658,  p<.05), 
and  distance  (F2i7=103.017,  pc. 001).  The  mean 
velocity  profiles  per  device  and  distance  can  been  seen 
in  figure  3.  Also  an  effect  of  subjects  was  found 
(F[  8=869.12,  pc. 01).  Post  hoc  tests  showed  that  the 
influence  is  as  follows: 

(1)  distance:  the  further  away  the  nearer  the  peak  (49%' 
for  the  farthest  target  and  58%  for  the  nearest) . 

(2)  device:  the  higher  the  friction  of  the  device  the 
further  away  the  peak  is.  With  the  optical  trackball 
having  the  lowest  friction,  the  peak  is  at  49%  and  the 
mechanical  force  feedback  trackball,  having  the 
highest  friction,  the  peak  is  at  59%). 

Direction  did  not  (significantly)  affect  the  relative 
distance  of  the  peak  velocity. 


Figure  3.  Velocity  profiles  of  the  mechanical 
mouse  (dashed  line),  the  optical  trackball 
(solid  line)  and  the  mechanical  force  feedback 
trackball  (dotted  line). 


3.4.  Target  resolution 

Assuming  variability  for  the  nearest  targets  of  10.6 
pixels  at  50%  of  the  distance,  a  prediction  accuracy  of 
69%  is  reached  within  7,5  degrees.  The  farthest  targets, 
having  a  mean  variability  of  14.6  pixels,  result  in  a 
69%  accuracy  within  4,5  degrees.  Taking  the  shortest 
distance  as  the  worst  case,  targets  must  be  at  least  15 
degrees  (2  times  7,5)  away  from  each  other. 

4.  Discussion  and  conclusions 

Since  curvatures  are  not  greater  than  the  amount  of 
variability  it  is  not  necessary  to  compensate  for  it  in  the 
general  prediction-algorithm.  This  means  that  the 
direction  of  the  users’  target  can  be  assessed  from  the 
direction  of  the  initial  movement  of  the  cursor. 

The  accuracy  of  this  prediction  will  depend  on  the 
object  resolution  and  on  the  relative  distance  whereon 
the  prediction  is  done.  In  earlier  research  it  was 
concluded  that  the  variability  climbs  to  a  maximum  at 
80%  of  the  distance  and  then  falls  down  rapidly  (see 

[3]). 

To  increase  the  accuracy  of  the  prediction  of  the 
direction  one  could  decrease  the  distance  between 
objects  or  change  the  distance  at  which  a  direction 
prediction  is  made.  Also,  multiple  predictions  can  be 
made  along  the  path. 

The  distance  of  the  target  can  be  assessed  by 
multiplying  the  distance  of  the  peak  velocity  by  a 
factor  two.  An  algorithm  per  device,  however,  would 
be  preferable.  The  multiplication  factors  would  then  be 
100/49,  100/53  and  100/59.  A  further  specialisation  is 
possible  if  the  factor  changes  when  the  distance  of  the 
peak  changes. 

For  each  variable  the  best  algorithm  would  be  a 
subject  specific  one.  It  depends  on  purpose  and  costs 
whether  this  is  rewarding. 

The  most  important  conclusion  to  be  drawn  is  that 
creating  a  useful  prediction  algorithm  seems  to  be 
possible.  A  prediction  of  the  target  distance  is  possible 
when  the  cursor  is  about  halfway.  A  prediction  of  the 
correct  direction  appears  possible  in  an  even  earlier 
stage. 
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Abstract 

The  interaction  techniques  (ITs)  that  are  used  for 
object  selection/manipulation  are  very  important  for 
easier  and  faster  user  interaction  with  objects  in  vir¬ 
tual  environment  (VE).  Although  research  in  this  area 
is  very  intensive,  few  works  are  interested  in  possibil¬ 
ity  to  combine  these  techniques  with  force  feedback.  In 
this  paper  we  present  short  review  of  ITs  for  immer¬ 
sive  VE.  We  discuss  the  possibilities  to  apply  these  ITs 
in  non-immersive  virtual  environment  with  force  feed¬ 
back.  We  also  show  advantages  and  disadvantages  of 
force  feedback  in  VE  and  vie  try  to  adapt,  the  methods 
to  provide  with  better  interaction. 

Keywords:  3D  interaction  techniques,  3D  user  inter¬ 
face,  virtual  environment,  selection  and  manipulation 
techniques,  haptics,  force  feedback,  PIIANToM 

1  Introduction 

To  perform  selection  and  manipulation  in  virtual 
reality  is  not  easy,  because  we  are  restricted  to  our 
arms  reach.  It  means  that  we  are  aide  to  select  and 
manipulate  objects  that  are  within  notably  limited 
area  in  the  neighbourhood  to  the  user.  Hence  to  real¬ 
ize  these  operations  it  is  necessary  to  use  interaction 
techniques  that  would  allow  us  to  accomplish  given 
type  of  interaction  better  and  faster. 

In  the  last  five  years,  the  research  of  interaction 
techniques  for  object  selection  and  manipulation  for 
immersive  VE  was  very  progressive.  But  we  have  not 
found  any  work  which  tried  to  combine  these  tech¬ 
niques  with  force  feedback  and  use  it  in  VE  applica¬ 
tions.  Adding  force  feedback  enables  users  to  feel  the 
object  shape  and  its  other  properties  such  as  hardness, 
lubricity  or  weight  of  the  object. 


In  this  paper  we  deal  with  selection/manipulation 
techniques  enhanced  with  force  feedback.  We  espe¬ 
cially  study  their  properties  i.e.  advantages  and  disad¬ 
vantages  which  occur  in  the  presence  of  haptics.  With 
respect  to  the  features  they  show,  we  adapt  them  to 
achieve  the  improvement  of  their  properties. 

For  realization  of  force  feedback  we  have  used  PHAN- 
ToM  1.0  device.  This  device  provides  single  3DOF 
force  feedback  applied  at  space  point  in  limited  space. 

2  Previous  work 

The  research  of  interaction  techniques  and  their 
modifications  was  focused  to  immersive  virtual  real¬ 
ity  (VR),  and  later  also  to  augmented  VR.  Several 
studies  [2,  5]  compared  individual  techniques  one  to 
another.  These  studies  allow  the  user  to  choose  ap¬ 
propriate  technique  for  given  type  of  interaction.  Un¬ 
fortunately,  only  a  few  of  them  dealt  with  haptic  in¬ 
teraction  in  VE. 

That  was  the  reason  why  we  chose  to  evaluate  the 
interaction  techniques  with  haptics  as  described  bel¬ 
low. 

3  Interaction  techniques 

VE  Interaction  techniques  extend  user’s  workspace 
for  object  manipulation  within  virtual  scene  and  they 
enable  to  select  the  object  situated  out  of  our  physical 
reach.  Important  requirement  for  these  methods  is 
that  they  must  be  easy  to  use  and  learn. 

The  taxonomy  of  interaction  techniques  for  manip¬ 
ulation  was  published  by  Poupyrev  et  al  [5].  Using 
this  taxonomy  we  study  the  usage  of  haptics  within 
egocentric  ITs,  especially  techniques  based  on  virtual 
hand  and  virtual  pointer  metaphor. 
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3.1  The  taxonomy  of  manipulation  tech¬ 
niques 

The  taxonomy  [5]  (Figure  1)  divides  current  ma¬ 
nipulation  techniques  for  VE  according  to  their  basic 
intera  ction  metaphors . 

E  MANIPULATION  TECHNIQUES 

—  Exocentric  metaphor 

| _ |  Worl  In  Miniature 

I  Automatic  scaling 

—  Egocentric  metaphor 

| —  Virtual  Hand  metaphor 

"Classical"  virtual  hand 
Go-Go 

Indirect  Go-Go 

I —  Virtual  Pointer  metaphor 
Ray-casting 

_ Flashlight 

Aperture 
Image  plane 

Figure  1:  Classification  of  VE  manipulation  tech¬ 
niques  depending  on  their  underlying  metaphor.  Re¬ 
produced  from  [5] . 

The  techniques  are  divided  into  exocentric  and  ego¬ 
centric  group.  Exocentric  techniques  arc  also  known 
as  God’s  eye  viewpoint,  where  user  interacts  with  a 
scene  from  outside.  An  example  is  World  In  Minia¬ 
ture  (WIM)  technique  [G]. 

On  the  contrary,  with  egocentric,  techniques  the 
user  is  situated  and  interacts  inside.  VE  space  shared 
by  user  and  objects.  These  techniques  are  subdivided 
according  to  the  metaphor  they  employ  -  either  vir¬ 
tual  hand  metaphor  (the  user  selects  or  manipulates 
the  object  using  a  virtual  representation  of  a  hand) 
or  virtual  pointer  metaphor  (where  user  interacts  by 
pointing) . 

3.2  Egocentric  techniques  for  selection  and 
manipulation 

In  this  section  we  describe  the  techniques  that  we 
used,  other  techniques  are  mentioned  only  briefly. 

3.2.1  Virtual  hand  metaphor 

With  methods  based  on  virtual  hand  metaphor  it 
is  possible  to  select  the  object  in  the  case  of  intersec¬ 
tion  between  the  object  and  a  cursor  (in  our  experi¬ 
ment,  the  cursor  follows  the  position  of  end  point  of 
PHANToM’s  arm).  These  methods  are  subdivided  to 


classical  virtual  hand  technique  and  techniques  that 
extend  arm  length  in  some  manner. 

The  virtual  hand  technique  uses  direct  linear  map¬ 
ping  of  physical  cursor  motion  to  motion  of  his  vir¬ 
tual  representation,  mostly  in  1:1  correspondence.  It 
means,  that  selection  and  manipulation  is  realized  in 
the  same  way  as  in  the  real  world. 

On  the  contrary,  techniques  extending  arm  length 
use  either  non-linear  mapping  or  arm  length  is  modi¬ 
fied  ly  some  signal.  The  Go-Go  and  Fast  Go-Go  in¬ 
teraction  techniques  use  non-linear  mapping.  The  Go- 
Go  technique  [4]  defines  a  local  region  around  the  user 
to  distance  D  (Figure  2).  While  the  physical  cursor 

Rv 


Figure  2:  Example  of  mapping  function  F,  for  Go-Go 
technique.  Reproduced  from  [4], 

stays  in  this  region,  it  uses  linear  mapping.  When  the 
cursor  leaves  this  region,  virtual  representation  of  the 
cursor  begins  to  move  faster  due  to  non-linear  map¬ 
ping.  This  way  we  extend  range  of  manipulation  at  the 
expense  of  manipulation  preciseness.  Modification  of 
this  method  is  the  Fast  Go-Go  technique  [1]  which  has 
no  local  region  and  applies  more  rapidly  growing  map¬ 
ping  function.  Next  there  belongs  the  Stretch  Go-Go 
technique  that  extends  arm  length  according  to  which 
region  around  the  user  the  cursor  is  located  in.  On  the 
contrary,  the  Indirect  stretch  Go-Go  technique  realizes 
the  change  of  length  using  two  signals. 

3.2.2  Virtual  pointer  metaphor 

With  the  virtual  pointer  metaphor,  selection  and 
manipulation  is  realized  by  a  ray  (ray-casting  tech¬ 
nique)  [1,3].  The  ray  is  defined  by  its  position  and  ori¬ 
entation.  Object  selection  and  grabbing  is  confirmed 
by  the  user  using  some  signal.  This  technique  has  spe¬ 
cific  disadvantages,  especially  selection  of  very  small 
or  distant  objects  is  complicated. 
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This  disadvantage  is  solved  using  technique  colled 
flashlight  1  [8].  The  user  ran  select  objects  that  are 
located  within  spotlight’s  came.  To  select  among  more 
objects  contained  in  the  cone,  it  is  possible  to  use;  an 
aperture  technique  [9]  (Figure  3).  It  is  modification 
of  flashlight  technique  whore  the  selection  ol  objects 
within  the  cone  is  done  using  a  collision  plane  (repre¬ 
sented  by  aperture  circle  here). 


Figure  3:  Aperture  technique  (Forsberg,  el  al.  1996). 

3.2.3  Manipulation  with  egocentric  techniques 

To  manipulate  object,  only  some  methods  based  on 
virtual  hand  metaphor  arc;  appropriate.  Especially  the 
classical  virtual  technique,  the  Go-Go  and  Fast  Go-Go 
technique  [1, 4].  With  the  St  retch  Go-Go  and  Indirect 
stretch  Go-Go,  the  manipulation  is  not  so  easy. 

From  methods  based  on  virtual  pointer  metaphor, 
the  ray-casting  technique  [I,  3]  is  used  to  change  ob¬ 
ject  location.  During  manipulation,  the  ob  ject  is  being 
relocated  using  the  ray.  The  change  of  position  can  be 
done  relatively  easily,  but  object,  rolat  inn  is  rather  dif¬ 
ficult  because  we  can  rotate  ob  ject  easily  only  around 
a  ray  axis.  Also  during  manipulation  it  is  suitable 
to  use  fishing  reel  technique  2  [1]  that  allows  to  pull 
and  push  objects  along  the  ray.  It.  is  realized  using 
two  buttons.  The  rotation  around  another  axis  can 
be  enabled  using  additional  signals. 

4  The  experiments 

In  our  experiments  we  tested  the  properties  ol  in¬ 
teraction  techniques  with  haptics  in  immersive  VL. 
Our  goal  was  to  find  out  if  these  methods  allow  users 
to  interact  better  arid  faster  with  the  virtual  scene. 
We  also  wanted  to  discover  what  problems  they  bring 
along  and  how  we  can  handle  them. 

The  experiments  used  the  virtual  environment  with 
several  simple  objects,  such  as  cube  and  sphere,  and 

1  This  rnothod  is  also  known  as  spot  Si^in  05  also  rone-casting 
loch  11  i  quo 

2 This  technique  is  also  known  as  ray-rjiFt ino,  with  rolling. 


PIIANToM  stylus  was  represented  by  a  virtual  sty¬ 
lus  (Figure  4).  Position  and  orientation  was  derived 
from  the  real  stylus  location.  Using  scene  geometry 
and  collision  detection  application  computes  appro¬ 
priate  force  arid  applies  force  feedback  to  user’s  finger 
in  PIIANToM.  All  computations  were  done  on  dual 
PC  system  with  R.T  Linux. 

Thus,  with  the  virtual  hand  metaphor,  we  can  touch 
arid  feel  the  objects  within  the  scene  and  select  them 
at  end-point  of  the  virtual  stylus.  The  object  selection 
is  confirmed  when  the  user  touches  object  and  presses 
the  stylus  button  at  the  same  time. 


Figure  4:  Example  of  the  scene  and  the  virtual  repre¬ 
sentation  of  PHANToM  stylus. 

With  virtual  pointer  metaphor,  the  location  of  ray 
was  determined  by  the  position  of  the  real  stylus  and 
direction  was  derived  from  its  orientation.  Thus  the 
ray  is  ousted  from  the  virtual  stylus  and  is  visualized 
by  a  short  segment.  In  this  case,  the  user  performs 
object  selection  by  intersecting  some  object  with  the 
ray.  When  feeling  haptic  contact,  user  may  confirm 
selection  by  pressing  the  stylus  button. 

4.1  Properties  of  selection  techniques  with 
force  feedback 

The  selection  task  can  naturally  exploit  contact  in¬ 
formation  provided  with  force  feedback.  It  helps  to 
detect  a  contact,  with  object  in  a  faster  and  more  con¬ 
vincing  way.  Here  we  show  various  approaches  of  force 
feedback  application,  together  with  assets  and  prob¬ 
lems  they  bring  along. 

4.1.1  Classical  virtual  hand  technique 

This  is  a  typical  technique  which  is  used  for  se¬ 
lection  of  objects  most  often.  Here  force  impact  is 
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realized  in  the  moment.,  when  it  conies  to  collision  be¬ 
tween  cursor  and  object.  Repulsive  force  is  applied  by 
PHANToM  device  to  simulate  natural  haptic  feeling. 

The  advantage  of  this  approach  is,  that  it-  allows  the 
user  to  explore  the  scene  in  a  given  scale  factor  (de¬ 
pending  on  adopted  linear  mapping).  Subsequently, 
the  user  is  able  to  create  a  partial  visualization  of  a  real 
shape  of  the  individual  objects.  Unfortunately,  with 
this  method  we  are  restricted  only  t.o  space  mapped 
to  physical  PHANToM  space,  accordingly.  Moving  in 
larger  virtual  space  using  suitable  scaling  we  achieve 
faster  motion  of  the  cursor  but  we  disallow  precise 
manipulation  in  the  whole  scene. 


X 

[0,  0, 0] 


fast  motion 


•  real  point  position  o  transformed  position 


Figure  6:  Fast  skidding  along  the  surface  that  is  due 
to  usage  of  the  Go-Go  technique. 


4.1.2  Go-Go  techniques 

The  disadvantage  of  the  classical  virtual  hand  tech¬ 
nique  is  partly  solved  using  Go-Go  techniques  [1,  4], 
When  the  cursor  is  located  in  the  local  region  this 
method  has  the  same  properties  as  previous  one. 

When  the  cursor  enters  outer  region,  a  user  can 
move  deeper  into  the  scene  using  non-linear  mapping. 
The  disadvantage  is  that,  due  to  the  non- linear  map¬ 
ping  the  distances  get  shorter,  and  object  shapes  ap¬ 
pear  deformed  (Figure  5).  Another  problem  associ- 


Figure  5:  Shape  deformation  of  sphere  using  Go-Go 
technique. 

ated  with  this  technique  shows  up,  when  cursor  ap¬ 
proaches  the  borders  of  the  maximal  reach.  When 
it  collides  with  a  distant  object  close  to  the  border, 
the  interaction  with  it  is  very  sensitive  and  unstable. 
Force  feedback  loop  detects  faster  and  deeper  pene¬ 
tration  of  cursor  in  an  object,  resulting  in  enormous 
force  grows,  user’s  reaction  and  unpleasant  vibrations. 
This  undesirable  property  can  be  partly  eliminated  us¬ 
ing  appropriate  force  filter  included  to  interaction  loop 
(e.g.  quadratic  filter). 

The  next  problem  is  occurrence  of  the  so  called 
fast  skidding  along  an  object  surface  when  the  cur¬ 
sor  moves  into  the  scene  (Figure  6).  This  problem  is 
notable  especially  in  the  case  of  faster  motion.  We 
tried  to  solve  these  problems  by  establishing  the  lo¬ 


cal  region  surrounding  every  object  within  the  scene. 
When  the  cursor  is  located  in  such  a  local  region,  we 
apply  linear  mapping.  Together  with  it  we  create  a 
mild  force  barrier.  This  purposely  created  artificial 
force  field  holds  the  user  close  to  the  object  and  it 
helps  him  to  not  leave  the  object  region  unknowingly. 
Of  course,  it  means  that,  user  would  feel  this  force  field 
and  we  experiment  with  its  proper  setting.  With  this 
approach,  the  user  is  able  to  recognize  the  real  object 
shape,  and  the  effect  of  the  shape  skidding  is  almost 
eliminated ,  Local  region  with  force  field  also  improves 
object  manipulation  (see  section  4.2.2). 

However,  this  solution  has  also  some  drawbacks. 
The  problem  appears  when  the  cursor  leaves  the  local 
region  of  an  object  and  non-linear  mapping  is  again 
switched  on.  To  change  mappings  immediately  could 
lead  to  the  situation  when  the  cursor  is  relocated  to 
the  inside  of  some  other  object.  Subsequently  repul¬ 
sive  force  changes  very  rapidly.  This  problem  may  be 
solved  using  an  algorithm  that  realizes  smooth  transi¬ 
tion  between  both  types  of  mappings. 

4.1.3  Ray-casting  technique 

With  ray-casting  technique  [1,  3]  there  are  two  ob¬ 
vious  solutions  how  to  apply  the  force  feedback.  The 
first  way  is  same  as  with  virtual  hand  metaphor  where 
the  ray  repulsion  from  the  object  is  used.  Force  is  ap¬ 
plied  in  point  that  is  the  centroid  of  the  ray  intersec¬ 
tions  with  an  object.  Situation  is  shown  in  Figure  7a. 
Using  this  solution,  the  user  is  partly  able  to  recognize 
a  profile  of  the  object  shape. 

The  second  way  is  to  derive  the  force  acting  against 
a  ray  motion  inside  the  object  (Figure  7b).  When  en¬ 
tering  object  with  ray  the  user  perceives  the  feeling  of 
steering  some  volume  with  liquid.  This  way  he/she  de¬ 
tects,  when  scanning  the  scene,  the  objects  with  which 
ray  collides.  But  in  the  quiescent  state  of  the  ray  we 
do  not  feel  any  force  resistance  even  in  the  collision 
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Figure  7:  Applying  the  force  when  ray  collides  with  an 
object,  (a)  Ray  repulsion  from  the  object,  (b)  Force 
acting  against  a  ray  motion  inside  the  object. 

situation.  We  have  solved  this  drawback  by  applica¬ 
tion  of  a  small  force  in  direction  opposite  to  the  ray. 
Now  we  can  easily  detect  the  collision  with  an  ob¬ 
ject  even  in  the  quiescent  state.  In  both  approaches 
rapid  force  changes  may  occur,  which  results  in  vibra¬ 
tions.  It  happens  in  the  cases  when  ray  collides  with 
some  distant  object.  Only  a  small  angular  change  in 
ray’s  direction  results  in  very  fast  penetration  speed 
and  unexpected  force  jumps.  Here  we  also  must  use 
sophisticated  filter,  or  rather  an  algorithm  which  re¬ 
duces  these  vibrations. 

4.1.4  Flashlight 

With  flashlight  method  [8]  we  can  use  both  pre¬ 
viously  mentioned  forms  of  force  feedback,  i.e.  ray 
repulsion  and  force  acting  against  a  ray  motion  inside 
the  object.  When  using  ray  repulsion  feedback  it  may 
be  difficult  to  approach  some  objects  from  some  loca¬ 
tions.  To  allow  easy  movement  in  densely  populated 
space  we  have  to  either  apply  only  a  small  repulsive 
force,  or  to  narrow  cone  apex  angle. 

The  second  approach  provides  better  and  a  more 
comfortable  solution.  Using  the  cone  we  may  select 
objects  easily  and  with  supplementary  force  feedback 
both  in  static  and  dynamic  form,  the  interaction  is  eas¬ 
ier.  As  in  previous  case,  the  force  have  to  be  applied 
carefully  to  prevent  ’’enemy  behaviour”  of  interaction 
loop. 

4.1.5  Aperture 

When  using  aperture  circle  for  selection  inside  cone 
space  the  second  method,  i.e.  force  acting  against  a 
ray  motion  is  the  preferable  solution.  Using  this  ap¬ 
proach  a  user  need  not  to  strain  when  passing  through 
the  objects  in  front  of  object  under  interest.  In  addi¬ 
tion,  we  have  used  two  levels  of  the  force  feedback. 


The  smaller  force  is  used  to  indicate  to  the  user  that 
some  objects  are  located  within  the  cone.  The  second, 
greater  force  signals  the  collision  of  some  object  with 
the  aperture  circle  (Figure  8).  Again,  the  size  of  these 
forces  has  to  be  chosen  carefully  so  that  a  user  rec¬ 
ognizes  it  but  is  not  restricted  in  interaction  freedom. 


Figure  8:  Usage  of  the  force  with  aperture  technique. 


4.2  Properties  of  manipulation  techniques 
with  force  feedback 

4.2.1  Manipulation  with  virtual  hand  technique 

The  advantage  is  that  this  technique  enhanced  with 
haptics  allows  the  user  to  relocate  the  object  faster 
and  with  great  precision.  Of  course  the  properties  of 
3D  environments  are  exploited  to  support  precise  po¬ 
sitioning.  On  the  contrary,  force  feedback  may  extend 
the  time  of  manipulation  when  we  want  to  relocate  ob¬ 
ject  in  the  dense  space.  This  situation  could  be  solved 
by  switching  force  feedback  off.  Unfortunately  we  lose 
haptic  contact  with  the  scene.  The  more  preferable 
solution  is  to  apply  smaller  forces  appropriate  for  this 
situation. 

4.2.2  Go-Go  techniques 

Manipulation  using  Go-Go  techniques  [1,  4]  with 
haptics  meets  the  same  problems  as  simple  virtual 
hand  technique.  Due  to  non-linear  mapping  especially 
in  farther  part  of  the  workspace  it  is  more  difficult  to 
achieve  the  precise  location  of  the  object  (mostly,  this 
problem  occurs  when  we  move  one  object  close  to  an¬ 
other).  Here  we  can  again  establish  the  local  region, 
defined  in  the  section  4.1.2,  around  all  objects,  and  to 
perform  more  precise  location  of  the  object. 

4.2.3  Ray-casting  technique 

Manipulation  with  this  technique  displays  the  simi¬ 
lar  properties  as  previous  techniques,  but  in  addition, 


117 


Eurohaptics  2t>01  Birmingham  July  2001 


another  problems  appear  as  in  the  case  of  selection. 
The  main  problem  is,  that  a  small  change  of  the  ray 
orientation  results  in  a  big  change  of  distant  object  po¬ 
sition.  Again  we  can  partly  solve  this  problem  using 
appropriate  force  filter  which  reduces  vibrations. 

The  next  question  is,  how  to  grab  an  object  by 
ray.  The  first  solution  is  to  transfer  the  scene  so  that 
the  ray  intersecting  the  selected  object  is  in  its  cen¬ 
ter.  The  second  possibility  is  to  move  the  object  in 
the  plane  orthogonal  to  ray,  and  to  pull  the  object  to 
ray.  This  choice  brings  along  the  problem  of  the  unex¬ 
pected  penetration  of  selected  object  into  another  one, 
after  grabbing  it.  To  avoid  unexpected  force  jumps,  we 
use  dynamically  growing  repulsive  force  which,  start¬ 
ing  from  zero,  subsequently  resolves  colliding  situa¬ 
tion.  And  finally,  the  third  solution  we  can  use,  is  to 
allow  grabbing  the  object  at  any  place. 

Only  the  third  solution  seems  to  be  optimal,  be¬ 
cause  it  allows  us  to  choose  any  rotation  axis  and  to 
manipulate  with  objects  quite  freely. 

5  Conclusions  and  future  work 

In  this  paper  we  show  that  well  known  interaction 
techniques  for  immersive  VE  can  be  naturally  com¬ 
bined  with  haptics.  Force  feedback  may  improve  the 
user’s  interaction  in  virtual  worlds.  Techniques  en¬ 
riched  with  haptics  support  faster  interaction  with  the 
objects.  On  the  other  hand  their  usage  also  brings 
many  problems.  Wc  mentioned  some  partial  solu¬ 
tions  of  these  problems,  but  many  others  still  remain 
opened. 

In  the  future  we  want  to  find  more  general  solutions 
of  problems  related  to  combination  of  two  interaction 
spaces.  The  first  one  is  the  visual  space  of  immersive 
VE  with  non-linear  coordinate  system  and  the  sec¬ 
ond  one  is  the  haptic  space  with  a  different  and,  as 
we  found,  even  dynamically  changing  coordinate  sys¬ 
tem.  There  are  also  many  other  techniques  that  we 
want  to  adapt  for  haptic  VE  (such  as  the  WIM  tech¬ 
nique  [6],  Voodoo  Dolls  [7],  etc.).  We  also  plan  to 
continue  in  the  search  for  novel  interaction  techniques 
for  VE  that  would  allow  us  to  interact  better  with 
VE.  Long  distance  goal  focuses  on  the  ways  of  inter¬ 
action  that  would  be  appropriate  for  visually  impaired 
people. 
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Abstract 

Investigating  cyclic  vertical  arm  movements  with  a  hand-held  load  in  an  airplane 
undergoing  parabolic  flight  profiles  allowed  us  to  study  how  humans  modulated  their  grip 
force  (GF)  when  the  gravitational  and  the  inertial  components  of  the  load  force  (LF)  varied 
independently.  Eight  subjects  participated  in  this  study;  four  had  already  experienced 
parabolic  flights  (ES)  and  four  were  inexperienced  (NES).  They  had  to  move  continuously  an 
instrumented  object  up  and  down  in  three  different  gravitational  conditions  (1  g,  1.8  g,  0  g).  In 
1  g,  GF  precisely  anticipated  the  fluctuations  of  LF  which  was  maximum  at  the  bottom  of  the 
trajectory  and  minimum  at  the  top  of  the  trajectory.  However,  as  the  subjects  incompletely 
released  their  grip  when  LF  was  minimum,  the  grip-load  force  relationship  was  not  linearly 
proportional  as  frequently  described  in  the  literature.  When  the  gravity  changed  (0  g  and  1.8 
g),  the  grip-load  force  coupling  persisted  for  all  the  subjects  from  the  first  parabola.  In  0  g,  GF 
was  accurately  adjusted  to  the  two  peaks  of  LF  occurring  at  the  two  extremities  of  the 
trajectory  due  to  the  absence  of  weight.  While  the  level  of  GF  modulation  was  immediately 
adapted  to  the  new  force  field  for  the  ES,  the  NES  dramatically  increased  their  grip  when  he 
was  faced  with  altered  gravity  for  the  first  time.  A  progressive  release  of  the  grip  occurred  and 
a  continuous  grip-load  force  relationship  with  regard  to  1  g  was  established  after  the  fifth 
parabola.  A  new  gravitational  field  was  rapidly  integrated  into  internal  CNS  models  which 
could  then  be  reused  as  required  by  the  occasion. 
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Abstract 

In  the  experiment  we  investigated  the  reaching  behav¬ 
iour  of  10-year-old  children  and  adults  for  virtual  objects 
moving  either  on  a  predictable  (linear)  path  or  an  unpre¬ 
dictable  path  ( with  random  direction  changes).  24  adults 
and  24  10-year-olds  observed  a  spherical  object  which 
varied  in  the  predictability  of  motion  (linear  vs.  nonlin¬ 
ear)  and  the  speed  of  object  motion  (17  cm/s  vs.  24  cm/s). 
In  an  action  condition  the  participants  had  to  reach  for 
the  objects  via  a  force-feedback  device.  We  compared 
reaching  movements  for  the  linearly  moving  object  with 
reaching  movements  for  the  nonlinearly  moving  object 
(but  only  those  trials  were  evaluated  where  the  normally 
unpredictable  object  moved  linear).  We  analysed  initial 
movement  direction  and  maximum  speed  of  the  first  sub¬ 
movement.  In  a  judgement  condition  participants  indi¬ 
cated  the  initial  movement  direction  on  a  rating  scale. 
Children  and  adults  produced  higher  initial  movement 
speeds  and  the  initial  movement  direction  was  geared 
further  towards  the  current  position  of  the  object  for  un- 
predictably  moving  objects.  Our  results  show  that  pre¬ 
dictable  and  unpredictable  behavior  is  processed  differ¬ 
ently. 

1.  Introduction 

We  investigated  the  visuomotor  control  of  children  and 
adults  reaching  for  an  object  that  was  displayed  on  a  com¬ 
puter  monitor  and  moved  either  predictably  or  unpre- 
dictably.  In  various  interception  tasks,  reaching  for  line¬ 
arly  moving  objects  has  been  studied  by  researchers  inter¬ 
ested  in  human  motor  control  (see  Schmidt,  1988)  and  the 
development  of  perceptual-motor  skills  (see  Keogh  & 
Sugden,  1985;  Dorfman,  1977;  Williams,  1973). 

Cognitive  aspects  of  interception  skills  have 
rarely  been  investigated  in  detail  in  this  field  of  research. 
Recent  infant  studies,  mostly  conducted  by  von  Hofsten 


and  his  group  (von  Hofsten,  1980,  1982,  1983;  von  Hof¬ 
sten,  Visbton,  Spelke,  Rosander,  &  Feng,  1998),  consti¬ 
tute  an  exception.  This  research  has  shown  that  young 
infants  are  capable  of  manually  intercepting  a  (slowly) 
moving  object  as  soon  as  they  begin  to  reach  for  station¬ 
ary  objects  (von  Hofsten  &  Lindhagen,  1979)  and  that 
their  rudimentary  interception  skills  improve  markedly 
between  four  and  eight  months  of  age.  Infants’  reaching 
movements  have  been  shown  to  be  predictive  in  that  they 
are  geared  toward  the  future  interception  point  rather  than 
at  the  object's  momentary  position  (von  Hofsten  1980; 
Clifton,  Muir,  Ashmead,  &  Clarkson,  1993).  Although 
infants  show  anticipation  of  a  future  interception  point  if 
the  object  moves  linearly,  reaching  is  interrupted  if  the 
object  moves  on  a  non-linear  path  (von  Hofsten  et  ah, 
1998). 

Reaching  for  linearly  moving  objects  has  also 
been  investigated  with  older  children  and  adults.  Reaching 
skills  seem  to  develop  quite  slowly  with  respect  to  accu¬ 
racy  and  interception  speed  and  are  not  fully  developed 
until  the  teenage  years.  Reaching  movements  of  adults 
have  been  shown  to  be  highly  adapted  to  object  speed: 
Objects  that  move  with  high  velocity  are  approached 
faster  than  objects  moving  more  slowly  even  if  partici¬ 
pants  are  instructed  to  reach  for  the  object  as  fast  as  pos¬ 
sible  (Smeets  &  Brenner,  1995). 

But  how  does  the  reaching  movement  change  if 
object  motion  is  unpredictable  like  in  the  case  of  animate 
objects?  More  specifically,  do  adults  as  well  as  children 
adapt  their  movements  to  the  escape  behaviour  of  a  target 
object  and  gear  them  toward  the  object's  momentary  posi¬ 
tion  instead  of  toward  an  anticipated  interception  point? 
To  investigate  this,  we  conducted  an  experiment  where 
participants  (10-year-old  children  and  adults)  had  to  inter¬ 
cept  a  moving  object  displayed  on  a  computer  monitor 
that  moved  either  linearly  or  showed  an  escape  behaviour. 


1  Now  at:  Friedrich-Miescher-Laboratory  of  the  Max-Planck-Society 
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Participants  controlled  another  virtual  object  to  intercept 
the  target  object. 

2.  Experiment 

2.1  Method 

Twenty-eight  adults  and  26  10-year-old  children  took  part 
in  the  experiment.  Two  adults  and  one  child  had  to  be 
excluded  due  to  technical  problems  and  three  adults  as 
well  as  two  children  had  to  be  excluded  from  the  analysis 
because  they  did  not  follow  the  instructions  (i.e.,  they  did 
not  try  to  catch  the  ball  as  fast  as  possible).  The  data  of  23 
adults  (11  female,  12  male)  and  24  children  (12  female, 
12  male)  was  analysed.  The  average  age  of  the  adults  was 
23  years  and  8  month  (min:  19;  1 ,  max:  28;7)  and  of  the 
children  10  years  and  2  month  (min:  9;  1 ,  max:  1 1;3). 

Participants’  task  was  to  intercept  a  spherical  ob¬ 
ject  moving  on  a  computer  monitor.  In  order  to  intercept 
the  object  they  had  to  move  a  second  object  that  they  con¬ 
trolled  via  a  PHANToM™  haptic  interface.  The  target 
object  moved  with  constant  speed  (either  17  cm/s  or  24 
cm/s)  starting  from  the  left  or  right  border  of  the  monitor. 
In  a  linear  condition,  the  target  moved  on  a  straight  and 
horizontal  path.  In  a  non-linear  condition,  the  target  was 
programmed  to  show  an  escape  behaviour,  i.e.,  it  changed 
its  direction  (with  a  new  direction  at  a  random  angle  be¬ 
tween  75  deg  and  105  deg  to  the  left  or  right  of  the  former 
direction)  as  soon  as  the  distance  from  the  participant’s 
object  fell  below  a  randomly  chosen  minimal  distance. 

The  two  conditions  were  presented  separately  in 
two  experimental  blocks.  Object  speed  and  direction  were 
varied  from  trial  to  trial.  To  familiarise  participants  with 
the  respective  target  behaviour,  eight  practice  trials  were 


presented  at  the  beginning  of  each  block.  In  the  linear 
condition,  eight  experimental  trials  were  given;  in  the  non¬ 
linear  condition,  a  total  of  40  experimental  trials  were 
given  in  eight  of  which  the  target  moved  on  a  linear  path, 
like  in  the  linear  condition  with  no  change  of  the  direc¬ 
tion.  Only  these  eight  trials  were  analysed  and  compared 
with  the  eight  experimental  trials  of  the  linear  condition. 
The  participants’  catching  movements  were  measured 

with  the  PHANToM™  haptic  interface. 

This  action  task  was  followed  by  a  judgement  task  where 
only  the  target  speed  varied.  After  the  target  had  left  its 
home-field  (marked  by  a  circle  around  the  starting  posi¬ 
tion),  participants  had  to  indicate  the  most  suitable  angle 
for  the  initial  sub-movement  of  the  interception  response. 
Two  practice  trials  were  followed  by  four  experimental 
trials  for  each  the  linear  and  non-linear  condition  sepa¬ 
rately. 


Figure  1.  Linear  (upper  panel)  and  nonlinear  condition 
(lower  panel).  In  the  linear  and  non-linear  condition  the 
target  objects  are  a  blue  and  orange  sphere,  respectively. 
The  object  controlled  by  the  participant  was  a  purple 
sphere.  Its  starting  position  was  the  left  and  right  lower 
corner  for  a  homefield  of  the  target  on  the  left  and  right 
side,  respectively.  The  circular  line  around  the  starting 
position  of  the  target  sphere  indicated  the  home-field.  The 
angle  a  indicates  the  initial  direction  of  the  reaching 
movement.  The  distance  dmin  was  chosen  randomly  each 
trial;  it  indicates  the  distance  at  which  the  escape  behav¬ 
iour  started. 

2.2  Results 

Data  of  participants’  initial  direction  of  movement  (rela¬ 
tive  to  the  current  object  position,  calculated  for  the  first 
100  ms  following  the  movement  initiation)  and  maximum 
speed  of  the  first  sub-movement  of  each  trial  were  ana¬ 
lysed.  The  initial  movement  direction  provided  informa¬ 
tion  whether  participants’  movements  were  rather  geared 
toward  the  target’s  current  position  or  toward  an  antici¬ 
pated  interception  point.  In  the  judgement  task,  the  esti¬ 
mated  angles  were  analysed.  ANOVAs  were  performed 
for  the  action  and  judgement  conditions  separately  on  the 
within-factors  object  speed  and  predictability  of  object 
motion  (linear  vs.  non-linear  condition)  as  well  as  on  the 
between-factor  age. 

Figure  2.  Initial  direction  in  the  judgement  condition  for 
children  and  adults  (L  =  linear  condition,  N=  nonlinear 
condition,  17  and  24  indicate  the  target  speed  of  17  cm/s 
and  24  cm/sec) 

2.2.1  Judgement  -  Initial  direction.  In  the  linear  condi¬ 
tion  estimated  angles  were  larger  than  in  the  non-linear 
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condition,  F(l,  46)  =  9.11 ,  p  <  .01.  Participants  aimed 
further  ahead  in  the  linear  condition.  The  estimated  angles 
were  also  larger  for  the  fast  object,  F(l,  46)  =  5.08,  p  < 
.05.  Overall,  no  age  differences  were  found,  F(  1,  45)  = 
1.71,  p  =  .  20). 

2.2.2  Action  -  Initial  direction.  The  initial  angle  of  the 
interception  response  varied  with  the  predictability  of  the 
target’s  motion:  The  angle  was  larger  in  the  linear  condi¬ 
tion  than  in  the  non-linear  condition,  F(l,  45)  =  9.15,  p  < 
.01.  This  indicated  that  participants  rather  aimed  for  an 
anticipated  interception  point  in  the  linear  condition  but 
approached  the  target  object  more  directly  in  the  non¬ 
linear  condition.  Overall  the  initial  direction  did  not  differ 
significantly  for  the  high  and  low  target  speed,  F(\ ,  45)  = 
2,51,  p  >  0.1.  The  two  way  interaction  of  the  factors  ob¬ 
ject  speed  and  predictability  of  object  motion,  F(  1 ,  45)  = 
5.02,  p  <  .05,  as  well  as  the  three  way  interaction  with 
age,  F(l,  45)  =  5.91,  p  <  .05,  turned  out  to  be  significant, 
however.  The  initial  angle  tended  to  be  smaller  for  the  fast 
object,  in  contrast  to  the  judgement  condition.  Children 
did  not  show  this  effect,  however,  in  the  non-linear  condi¬ 
tion. 


Condition 
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Figure  3.  Average  produced  initial  direction  in  the  action 
condition  for  children  and  adults.  (L  =  linear  condition,  N= 
nonlinear  condition,  17  and  24  indicate  the  target  speed  of 
1 7  cm/s  and  24  cm/sec). 

2.2.3.  Action  -  Maximum  speed.  The  maximum  speed  of 
the  first  sub-movement  varied  with  both  object  speed  and 
the  predictability  of  object  motion.  It  was  higher  in  the 
non-linear  condition  than  in  the  linear  condition,  F(l,  45) 
=  55.23,  p  <  ,001,  and  higher  for  the  fast  object  than  for 
the  slow  object,  F(l,  45)  =  107.95,  p  <  .001.  Overall, 
adults  produced  higher  movement  speeds  than  children, 
F(l,  45)  =  5.70,  p  <  .05,  but  age  did  not  interact  with  the 
object  speed,  F  <  1  or  the  predictability  of  object  motion, 
F'(l,  45)  =  1.37,  p  =  .25. 
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tion  task  where  the  interception  point  can  be  anticipated 
and  approached  with  a  velocity  tuned  to  the  target  speed. 
In  the  case  of  unpredictable  objects  the  anticipation  of  an 
interception  point  is  not  possible  and  participants  tried  to 
reach  for  the  target  much  faster  and  more  directly  toward 
the  target. 

Based  on  the  present  results,  it  appears  possible  to 
investigate  (implicit)  knowledge  about  the  predictability 
of  object  motion.  In  future  studies,  children’s  knowledge 
about  animate  and  inanimate  objects  could  be  investigated 
using  our  paradigm.  Children  making  mistakes  in  judging 
whether  an  object  belongs  to  the  animate  or  inanimate 
category,  nevertheless,  may  show  different  reaching  be¬ 
haviour  when  asked  to  catch  these  objects. 


CHILDREN 

110-1 . . — 
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Figure  4.  Average  produced  maximum  speed  for  the  first 
sub-movement  for  children  and  adults.  (L  =  linear  condi¬ 
tion,  N=  nonlinear  condition,  17  and  24  indicate  the  target 
speed  of  17  cm/s  and  24  cm/sec). 


3,  Discussion 

In  general,  we  found  different  visuomotor  control  strate¬ 
gies  for  reaching  toward  targets  moving  predictably  or 
unpredictably.  The  reaching  movement  differed  with  re¬ 
spect  to  both  the  initial  movement  direction  and  maximum 
speed  of  the  first  sub-movement.  In  the  linear  condition, 
participants  aimed  at  an  anticipated  interception  point 
while,  in  the  non-linear  condition,  participants  approached 
the  object  more  directly.  Participants  reached  faster  for 
unpredictable  than  for  predictable  targets.  Additionally 
adults  and  children  reached  faster  for  the  fast  object  than 
for  the  slow  object. 

Taken  together,  these  results  show  that  partici¬ 
pants’  reaching  movements  were  adapted  to  both  target 
speed  and  predictability  of  the  target  motion.  Participants 
used  different  strategies  in  the  two  conditions.  In  the  lin¬ 
ear  condition,  reaching  for  the  target  is  a  simple  intercep- 
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Abstract 

Physical  models  continue  to  form  an  essential  outcome  from  industrial  design  practice.  Whist  the  professional  may  be 
removed  from  the  “hands-on  ”  model  build  by  employing  the  sendees  of  a  model-maker,  many  designers  consider  such 
physical  interaction  as  an  important  part  of  their  idea  development  process.  Visual  and  tactile  feedback  from  model 
generation  is  considered  by  many  as  an  essential  part  of  the  experience  of  physical  interaction  with  form  and  material. 
However,  the  advent  of  remote  model  building  technologies  via  rapid  prototyping  and  computer  controlled  machining,  has 
given  professionals  an  alternative  method  of  working  which  offers  other  advantages  to  the  design  process.  This  has  meant 
that  the  designer  is  increasingly  removed  from  such  physical  interaction  with  the  model. 

The  emergence  of  three  dimensional  digital  modelling  via  haptic  feedback  devices  may  offer  a  route  whereby  professional 
designers  can  gain  the  benefits  of  a  digital  process  yet  continue  to  be  involved  with  tactile  design  modelling. 
Acknowledging  the  need  to  utilise  digital  design  techniques,  this  paper  investigates  the  capabilities  of  haptic  modelling  for 
use  within  industrial  design  practice.  The  research  is  based  on  an  industrial  design  case  study  for  a  communication  device 
that  was  undertaken  by  the  authors. 

The  paper  discusses  the  attempts  made  to  emulate  workshop-based  modelling  activities  with  Sensable  Technologies 
Phantom  haptic  feedback  device  using  Freeform  software.  Whilst  the  initial  response  to  the  system  was  not  entirely 
satisfactory,  the  research  identified  the  potential  to  use  the  haptic  feedback  device  to  produce  forms  that  would  be  virtually 
impossible  to  specify  using  traditional  digital  techniques.  Having  gained  some  experience  in  the  capabilities  of  the 
hardware  and  software,  creative  opportunities  emerged  that  were  a  direct  result  of  the  application  of  this  technology. 


1.  Introduction 

As  a  profession  involved  in  the  definition  of  product 
form,  industrial  designers  make  extensive  use  of  three- 
dimensional  (3D)  models.  These  may  vary  in 
sophistication  from  relatively  simple  study  models  to  full 
prototypes  ([1]  Knoblaugh  1958  pl5,  f2]  Kojima  1991 
p38).  The  type  of  model  most  associated  with  industrial 
design  practice  is  the  appearance  model,  which  embodies 
the  form  of  the  production  item  but  none  of  the 
functionality  ([3]  Powell  1990  pi  1). 

Prior  to  the  advent  of  remote  model  building 
technologies  such  as  rapid  prototyping  and  computer 
controlled  machining,  models  would  be  produced  by 
manual  working  and  craft-based  techniques.  Some 
designers  take  design  decisions  as  they  manually 
manipulate  the  material,  modifying  it  accordingly. 


As  professional  practice  makes  increasing  use  of 
remote  model  building  technologies,  the  opportunity  for 
the  designer  to  be  directly  involved  in  the  shaping  of 
material  decreases.  This  has  been  shown  to  be  the  case 
with  the  full  range  of  models,  from  study  models  to 
prototypes  ([4]  Sharbaugh  in  Carrabine  1999  p24).  In 
contrast  the  digital  technology  that  enables  remote  model 
building  has  also  enabled  increasing  use  of  virtual 
evaluation  whereby  3D  computer  aided  design  (CAD) 
and  computer  aided  industrial  design  (CAID)  systems  are 
used  to  provide  photo-realistic  visualisation  and  real-time 
animation. 

One  could  argue  that  the  experienced  practitioner 
might  have  sufficient  levels  of  skill  and  judgement  to 
bypass  the  direct  interaction  with  form.  Indeed,  personal 
experience  has  shown  that  such  practice  often  takes  place 
when  deadlines  are  tight  or  resources  are  limited. 
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However  there  is  a  strong  ground.swcll  of  opinion  that 
tactile  product  development  is  beneficial  to  the  final 
products  form  and  so  a  way  should  be  found  to  combine 
the  craft  bused  techniques  with  digital  product 
development. 

Recognising  this  need  the  emerging  technology  of 
digital  modelling  using  a  haptic  feedback  device  may 
have  the  potential  to  develop  and  encourage  the  physical 
manipulation  of  form  on  a  virtual  level.  Haptic  feedback 
devices  give  the  operator  the  “feel”  of  the  virtual  object. 
If  the  operator  moves  a  cursor  onto  an  object  as  seen  via 
the  monitor,  they  actually  feel  its  presence  via  an  electro¬ 
mechanical  system  attached  to  the  pointing  device. 

The  authors,  having  backgrounds  as  both  design 
practitioners  and  educators,  have  explored  the  nature  of 
haptic  modelling,  considering  its  potential  for  integration 
into  industrial  design  practice  and  design  education. 
There  follows  a  record  of  a  programme  of  research  that 
involved  the  industrial  design  of  a  highly  conceptual 
communication  device  that  was  produced  as  an  entry  for 
the  Nagoya  International  Design  Competition  in  May 
2000.  The  structure  of  the  case  study  follows  the  three 
phases  of  professional  practice  as  identified  by  Pipes  [5], 
involving  concept  generation,  design  development,  and 
specification.  These  arc  now  explored  in  some  detail. 

2.  Concept  Generation 

The  brief  for  the  communication  device  specified  that 
its  form  should  cross  the  boundaries  between  what  we 
consider  to  lie  jewellery  and  a  consumer  product.  The 
design  was  to  have  some  of  the  functionality  of  a  mobile 
telephone,  without  the  transient  fed  of  a  polymer 
consumer  product.  Concept  generation  was  undertaken 
using  paper-based  sketching,  examples  of  which  can  he 
seen  in  Figure  1 . 

Whilst  CA1D  was  available  throughout  the  project,  the 
industrial  designer  felt  that  the  application  of  this 
technology  was  inappropriate  at  the  concept  generation 
stage  due  to  the  lack  of  spontaneity  afforded  by  its 
modelling  methods.  This  is  identified  b>  Pipes  [5]  when 
he  slates  that  “Conventional  CAD  at  an  early  stage  can 
stifle  creativity  by  its  insistence  that  the  designer 
provides  the  system  with  exact  dimensional  and 
geometric  information  right  from  the  start”. 

A  small  brooch-like  product  emerged  from  (he 
concept  generation  phase,  its  form  fused  around  an 
elliptical  body  with  a  large  answer  button.  Other 
functionality  was  accessed  via  three  smaller  buttons 
positioned  on  one  end.  The  spcaker/microphonc  was 
located  on  the  opposite  edge. 

The  paper-based  sketches  provided  the  industrial 
designer  with  sufficient  detail  on  Conn  and  size  to 
progress  to  the  second  phase  of  design  development.  If 


operating  to  a  more  traditional  design  methodology,  this 
phase  may  involve  some  modelling  in  “soft”  materials 
such  as  Styrofoam,  or  even  the  manipulation  of  more 
resistant  materials.  For  the  purpose  of  the  case  study, 
haptic  modelling  was  to  he  introduced  as  an  alternative. 


Figure  1 .  Concept  generation  using  paper-based 
techniques. 

3.  Design  Development 

Scnsable  Technologies  based  in  Boston  Massachusetts 
made  the  Phantom  Desktop  input  device  and  Freeform 
software  available.  Twin  Intel  450Mhz  processors  with 
512MB  of  RAM  were  used  to  run  the  software,  and 
Scnsable  Technologies  UK  consultant  provided  support. 
The  Phantom  Desktop  device  can  be  seen  in  Figure  2. 


Figure  2.  Phantom  Desktop  input  device  with 
Freeform  2  software. 


As  a  precursor  to  the  product  being  modelled,  an 
exploratory  use  of  the  technology  was  undertaken.  This 
involved  an  evaluation  of  the  additive  and  subtractive 
(Boolean)  modelling  techniques,  along  with  smoothing 
operations  and  manipulation  of  material  density.  The 
move  from  interacting  with  a  physical  object,  to  a  virtual 
model,  was  initially  found  to  be  an  unusual  experience. 
However,  as  familiarity  with  the  software  and  system 
developed,  the  modelling  capabilities  of  die  media 
emerged.  Indeed  it  soon  became  apparent  that  the  haptic 


modelling  system  was  capable  of  producing  forms  that 
could  not  be  generated  using  CAID,  although  the  value 
of  these  relatively  abstract  surfaces  represent  a  separate 
issue.  Figure  3  shows  one  of  the  models  produced  during 
the  exploration  of  the  media. 

After  a  period  of  familiarisation,  it  became  evident 
that  the  modelling  system  could  be  used  on  two  levels. 
The  first  involved  techniques  closely  associated  with 
normal  CAD  modelling,  whereby  forms  could  be 
generated  by  non-haptic  input  e.g.  creating  standard 
shapes  and  modifying  by  stretching.  The  second 
technique  was  true  haptic  modelling  which  had  the 
potential  to  produce  forms  via  various  shaping  operations 
whilst  receiving  physical  tactile  feedback. 

Following  the  evaluation  of  the  haptic  modeller, 
which  only  lasted  some  2-3  hours  (a  credit  to  the 
usability  of  the  system),  the  decision  was  taken  to  mode! 
the  communication  device  using  the  using  both  of  these 
methods.  This  was  particularly  appropriate  for  the 
communication  device,  as  the  basic  outer  form  of  the 
product  was  easily  constructed  from  primitive  shapes 
combined  via  Boolean  operations.  However  to  fulfil  the 
design  brief  in  removing  the  typical  feel  of  a  plastic 
injection  moulded  product  the  basic  form  was  intended  to 
be  enhanced  by  surface  finishes  more  normally 
associated  with  hand  crafted  products. 

The  basic  outer  form  was  first  created  in  the 
modelling  system  by  taking  a  sphere  and  distorting  it  to 
produce  the  required  flat,  curved  form.  Subtracting  a 
suitable  scaled  cylinder  created  the  central  cutout  for  the 
‘answer’  button.  The  speaker/microphone  notches  were 
to  be  modelled  by  haptic  feedback  techniques.  The 
designer  was  attempting  to  create  the  effect  one  would 
achieve  by  taking  a  scoop  of  soft  ice  cream,  leaving  a 
cavity  with  smooth  sides.  When  the  operator  performed  a 
haptic  scooping  operation,  the  surfaces  were  excessively 
rippled,  as  there  was  no  tight  control  of  the  motion.  This 
was  the  first  disappointment  in  the  use  of  the  modelling 
system,  which  will  be  discussed  later  in  this  paper.  The 
notches  were  again  produced  using  non-haptic  techniques 
of  subtracting  forms  generated  by  extruding  curves. 

At  this  point  it  was  recognised  that  the  form  produced 
had  been  developed  in  the  haptic  system  but  bad  not  used 
any  of  the  benefits  of  the  haptic  feedback.  The  system 
was  being  used  much  like  a  traditional  CAD  system.  As  a 
parallel  test  the  communication  device  was  modelled 
using  a  surface  modeller  familiar  to  the  operator  in  less 
than  Vi  an  hour  and  was  imported  into  the  Freeform 
software  via  an  STL  file  (a  standard  tile  transfer  format 
which  transfers  the  shape  information  via  triangular 
facets).  The  imported  surface  can  be  seen  in  Figure  4. 
This  raises  questions  about  the  suitability  of  haptic 
techniques  for  developing  certain  forms.  Again  this  will 
be  discussed  later. 


Figure  4,  Imported  CAID  surface. 


The  surface  finish  to  be  applied  to  the  model  was  not 
rigorously  defined  at  the  concept  generation  stage  as  it 
was  intended  to  explore  the  possibilities  of  forming  the 
virtual  material  using  techniques  closely  related  to  craft- 
based  interaction.  The  first  finish  to  be  explored  was  a 
hammered  finish  normally  applied  to  soft  metals  such  as 
copper.  The  desired  result  was  achieved  by  using  a 
rounded  tool  to  repeatedly  strike  the  surface.  It  was 
important  for  the  operator  to  understand  the  parameters 
that  could  be  varied  to  affect  the  material  properties  of 
the  model.  Using  a  soft  material  produced  very  smooth 
rounded  and  rather  indistinct  indentations  whereas  harder 
material  settings  produced  shapes  that  resembled  more 
closely  cuts  into  the  surface.  The  progression  of  the 
hammering  can  be  seen  in  Figure  5. 


Figure  5.  Haptic  modelling  of  hammered  effect. 

The  haptic  system  was  easily  able  to  take  advantage  of 
other  digital  design  techniques  seen  in  other  CAD 
systems.  For  example  the  hammer  finish  was  only 
applied  to  %  of  ihe  model  and  mirroring  reproduced  the 
other  quarters.  Careful  attention  was  paid  to  over  striking 
the  centre  lines  to  ensure  that  they  did  not  exhibit  any 
mirror  effects. 


When  completely  hammered,  the  surface  looked  quite 
effective  hut  the  edges  of  the  hammer  marks  were 
considered  too  sharp  so  the  whole  model  was  smoothed- 
out  to  soften  corners.  The  final  finish  can  be  seen  in 
figure.  6. 


Figure  6.  Final  hammered  effect. 


The  final  design  features,  the  ‘answer’  button  and 
three  function  buttons,  were  once  again  added  using 
traditional  CAD  design  techniques  based  on  standard 
primitive  shapes  and  extruded/revolved  curves.  I  he 
completed  model  was  saved  as  an  STL  file  and  exported 
to  a  (  AID  system  for  rendering.  The  rendered  product 
can  be  seen  in  Figure  7,  and  as  a  photomontage  with  user 
in  Figure  8. 


Figure  7.  Rendered  hammer  finish. 


4.  Specification 

The  specification  of  product  form  to  design  engineers 
and  manufacturers  who  arc  conversant  with  digital  design 
techniques  is  relatively  straightforward,  requiring  the 
transmission  of  digital  geometry  in  a  mutually 
compatible  format  e.g.  IOKS  or  STL. 

The  proposition  for  the  hammered  product  was  for  a 
low-volume,  jewellery-like  product.  It  would  therefore  be 
straightforward  to  output  the  surface  geometry  of  the 
body  to  an  STL  file  and  utilise  this  information  to 
produce  the  components  in  wax  using  a  rapid  prototyping 
system  such  as  Sanders.  Investment  casting  could  then  be 
used  to  manufacture  the  components  as  a  direct  copy  of 
the  model. 

5.  Conclusions 

This  design  project  has  raised  a  number  of  issues,  both 
positive  and  negative,  relating  to  haptic  modelling. 

The  ease  of  use  of  the  software  makes  it  very 
straightforward  for  even  inexperienced  computer 
operators  to  start  modelling  very  quickly.  The  software 
does  however  require  a  high  level  of  hand-eye 
coordination  for  the  tools  to  be  used  effectively.  This 
should  not  be  considered  as  a  problem  since  this  skill  is 
expected  of  designers  as  part  of  their  repertoire  and  so 
they  should  feci  quite  at  home  with  the  input  methods 
used.  It  docs  tend  to  alienate  other  groups  from  using  the 
software.  For  example,  experience  with  engineers 
attempting  to  use  the  system  has  shown  that  in  general 
they  lack  the  skill  to  operate  it  effectively  although  they 
appreciate  the  value  of  tactile  feedback. 

'J'hc  apparent  ease  of  use  should  not  disguise  the  fact 
there  is  a  technical  element  to  the  software  that  need  to 
be  understood.  This  is  especially  apparent  when  dealing 


27 


with  aspects  such  as  the  variable  hardness  and  ‘density’ 
of  the  material.  Even  in  the  relatively  small  modelling 
exercise  undertaken  for  this  project  it  was  necessary  to 
vary  material  properties  to  achieve  the  desired  results. 
Although  this  is  relatively  easy  to  achieve  in  terms  of  the 
user  interface  the  user  needs  to  understand  the  knock  on 
effects  such  modifications  have  on  the  model.  For 
example  reducing  the  ‘density’  of  the  model  enables  finer 
detail  to  be  produced  but  the  size  of  the  storage  space  for 
the  model  increases  dramatically  (100  Mb  files  are 
common)  and  even  using  relatively  powerful  computers 
performance  reduces  dramatically.  This  can  be  very 
frustrating  to  a  designer  who  does  not  want  to  be 
concerned  with  such  ‘technical’  details. 

At  the  beginning  of  the  project,  the  expectations  of  the 
haptic  modelling  system  were  high.  Assumptions  were 
made  that  the  modelling  techniques  would  be  very  close 
to  those  of  conventional  foam  and  clay  concept 
modelling.  Unfortunately,  the  functionality  of  the 
software  and  hardware  made  it  difficult  to  obtain  the 
surface  quality  needed  for  both  rendering  and  production. 
This  was  highlighted  by  the  attempts  made  to  produce  a 
clean  scoop  when  modelling  the  speaker/microphone 
detail.  These  shapes  were  easily  reproduced  by  other 
techniques  but  these  duplicated  more  traditional  methods 
of  modelling  found  in  traditional  CAD  systems  and  so 
did  not  benefit  from  haptic  feedback.  The  fact  that  the 
system  can  model  in  this  manner  is  advantageous  but  it  is 
disappointing  that  haptic  modelling  could  not  reproduce 
the  same  effects.  It  is  almost  an  admission  that  haptic 
techniques  are  not  yet  able  to  model  large  smooth 
surfaces  effectively. 

Another  related  area  that  concerned  many  operators  of 
the  system  was  the  reproduction  of  sharp  edges.  This  is 
highlighted  in  Figure  8,  which  shows  feathering  on  the 
edges  of  the  button  cutouts. 


Figure  8.  Edge  feathering. 


The  method  of  representing  the  mode!  in  the  Freeform 
software  means  that  edges  between  surfaces  can  never  be 
perfectly  sharp.  They  are  always  feathered  to  some 
extent.  This  feathering  can  be  reduced  by  increasing  the 
accuracy  of  the  model  (with  the  consequences  already 


mentioned)  but  will  always  be  present.  This  always 
concerns  operators  as  they  zoom  in  to  the  model  closely. 
In  practice  the  feathering  is  not  a  problem  as  when 
viewed  at  normal  scale  the  feathering  is  not  visible  and 
has  never  shown  up  in  derived  rapid  prototype  models. 

The  responsiveness  of  the  haptic  modelling  system  to 
the  generation  of  Ihc  hammered  effect  was  impressive. 
The  production  of  the  hammered  effect  was  very  closely 
associated  with  traditional  hands-on  modelling,  whereby 
the  designer  controlled  the  manipulation  of  material 
whilst  responding  to  tactile  feedback. 

For  industrial  design  practitioners,  the  findings  of  the 
case  study  indicate  that  haptic  modelling  as  currently 
implemented  in  the  Freeform  software  has  its  limitations. 
Whilst  the  stylistic  trends  within  industrial  design  require 
smooth,  crisp  surfaces,  the  authors  feel  that  haptic 
modelling  has  significant  limitations.  However,  the 
principle  of  haptic  feedback  being  used  in  the  model 
process  is  unquestionable  and  when  the  product  shapes 
being  designed  are  appropriate  then  the  Freeform 
software  and  haptic  modelling  are  the  only  realistic 
modelling  method.  The  only  alternative  method  for 
reproducing  the  hammered  finish  would  he  to  use 
greyscale  pictures  to  generate  a  bump  mapped  surface 
but  the  results  are  unlikely  to  be  as  effective.  One  can 
only  assume  that  future  development  in  the  hardware  and 
software  for  haptic  modelling  will  address  the 
shortcomings  identified  in  the  case  study.  Further  work 
will  then  he  required  to  re-evaluate  such  capabilities. 

In  terms  of  the  generation  of  random,  almost  craft 
effects,  haptic  modelling  has  much  to  offer.  The  authors 
feel  that  such  a  system  may  be  of  interest  to  practitioners 
in  may  areas  of  design  such  as  jewellery.  This  would  he 
particularly  relevant  if  the  virtual  material  could  emulate 
the  characteristics  of  precious  metals  that  may  be  too 
expensive  to  use.  Indeed  it  may  be  possible  to  use  haptic 
modelling  to  practice  on  a  virtual  material  before 
working  on  the  real  (and  expensive)  precious  metal. 
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Abstract 

This  -paper  describes  on-going  research  at  the  Swiss 
Federal  Institute  of  Technology  to  develop  new 
paradigms  for  interactive  segmental  ion  of  medical  3D 
volume  data.  In  our  segmentation  system,  we  are  us¬ 
ing  a  haptic  device  as  a  3D  mouse  with  force-feedback. 
The  paper  introduces  the  basic  components  of  our 
framework  and  the  ideas  behind  our  multi-modal  ap¬ 
proach.  On  the  one  hand  we  are  trying  to  increase 
the  sense  of  immersion  of  the  uses  by  alloying  him  to 
interact  in  a  natural  way  with  the  presented  data,  on 
the  other-  hand  we  are  using  force-feedback  as  an  addi¬ 
tional.  channel  to  mediate  information.  The  segmen¬ 
tation  of  linear  structures  in  the  human  body  serves 
as  a  case  study  for  our  system.  We  successfully  used 
our  approach  to  extract  the  skeleton  of  a  radiological 
colon  data  set,  which  then  screed  as  the  basis  for  a 
full  segmentation  of  the  object  of  interest. 

1  Introduction 

Over  the  last  decades,  computer  imaging  techniques 
have  become  an  important  tool  in  the  practice  of 
modern  medicine.  In  medical  imaging,  large  voxel 
data  sets,  containing  information  about  the  inter¬ 
nal  anatomy  and/or  physiology  of  a  living  patient, 
are  obtained  from  a  variety  of  tomographic  imag¬ 
ing  modalities,  such  as  Computer  Tomography  (CT) 
or  Magnetic  Resonance  Imaging  (MR1).  Before  any 


high-level  reasoning,  for  instance  radiat  ion  treatment 
planning,  can  be  applied  to  this  data,  it  has  to  be  bro¬ 
ken  down  into  its  major  structural  components.  The 
process  in  which  the  data  sets  are  subdivided  into 
their  constituent  parts  is  called  segmentation.  The 
algorithms  used  in  this  step  can  be  classified  with 
regard  to  their  degree  of  automation: 

•  Manual  segmentation: 

This  approach  is  highly  insensitive  to  noise,  tol¬ 
erant  of  incomplete  information  and  in  many 
cases  sufficiently  precise.  Nevertheless,  it  is  very 
time  consuming  and  has  to  be  done  in  a  tedious 
pixel-by-pixel  manner.  Furthermore,  it  lacks  re¬ 
producibility. 

•  Automat  ic  segmentation: 

This  term  refers  to  computer  algorithms  which 
can  segment  an  image  without  manual  interac¬ 
tion.  These  methods  can  only  be  applied  suc¬ 
cessfully  within  exactly  defined  bounds.  This  is 
mainly  due  to  the  difficulty  of  formalizing  the 
a  priori  knowledge  of  the  examined  anat  omical 
structures  as  well  as  incorporating  this  knowl¬ 
edge  into  the  segmentation  process.  Unfortu¬ 
nately,  none  of  the  so  far  published  algorithms  is 
robust  enough  to  provide  general  purpose,  fully 
automatic  data  segmentation. 


•  Semi-automatic  segmentation: 

In  between  the  former  two  approaches  lie  semi- 
automated  algorithms,  which  try  to  merge  the 
advantages  of  both  worlds.  Rather  than  at¬ 
tempting  to  duplicate  the  complex  and  poorly 
understood  human  capability  to  recognize  ob¬ 
jects,  they  try  to  provide  an  interactive  envi¬ 
ronment  in  which  users  control  the  segmentation 
process  and  exploit  their  expert  knowledge. 

The  developed  methods  can  be  further  classified 
according  to  the  underlying  interaction  paradigms  in 
either  two  or  three  dimensions.  In  a  two-dimensional 
approach  the  segmentation  is  done  in  a  slice-by-slice 
manner,  making  a  subsequent  reconstruction  step 
necessary,  in  which  the  previously  segmented  con¬ 
tours  are  combined  into  a  three-dimensional  struc¬ 
ture. 

More  seldom  are  systems  that  exploit  the  possibili¬ 
ties  of  interactive  3D  segmentation  ['13,  ]  7,  8,  19,  11]. 

This  is  often  justified  by  the  problems  that  arise 
due  to  adding  another  dimension  to  the  user  interac¬ 
tion: 

•  Editing,  controlling  and  interacting  in  three  di¬ 
mensions  often  overwhelms  the  perceptual  pow¬ 
ers  of  a  human  operator. 

•  Todays  desktop  metaphors  are  based  on  two- 
dimensional  interaction  and  can  not  easily  be 
extended  to  three  dimensions. 

•  The  visual  channel  of  the  human  sensory  system 
is  not  designed  for  the  perception  of  volumetric 
data. 

These  major  drawbacks  arc  valid  in  terms  of  in¬ 
teractive  systems  that  are  based  on  two-dimensional 
Window-Mouse-Icon-Pointer  (WIMP)  interfaces  that 
solely  rely  on  the  visual  sense  of  the  human  operator. 

In  the  presented  project  these;  limitations  are  alle¬ 
viated  by  enhancing  the  segmentation  process  with 
additional  sensory  feedback.  Properly  applied,  the 
use  of  more  than  one  sense  to  investigate  complex 
data  sets  promises  to  improve  the  user’s  understand¬ 
ing  of  the  data  and  their  ability  to  interact  with  them 


Possible  approaches  for  multi-modal  interaction 
stemming  from  virtual  reality  applications  are  the 
addition  of  auditory  as  well  as  haptic  feedback.  In 
this  work  we  focus  on  providing  additional  percep¬ 
tual  information  to  the  medical  expert  by  using  force- 
feedback  technology. 

2  Related  Work 

Though  the  need  for  understanding  the  influence  of 
human-computer  interaction  on  semi-automatic  seg¬ 
mentation  is  recognized,  only  very  few  research  has 
been  done  in  this  direction.  The  only  work  we  know 
of  investigates  this  problem  for  2D  segmentation  [  i  0] . 

Other  related  research  mainly  focuses  on  separate 
aspects.  New  3D  interaction  paradigms,  that  can  as¬ 
sist  the  end  user  in  navigating  in  volume  data  sets 
have  been  evaluated  in  [3,  19,  'Ll],  but  this  work  only 
focuses  on  visual  interaction  with  the  data.  Never¬ 
theless,  the  results  show,  that  novel  ways  of  inter¬ 
action  can  enhance  the  navigation  in  the  data.  The 
application  of  force-feedback  to  improve  the  under¬ 
standing  of  volume  data  sets  in  a  medical  setting  has 
been  done  in  the  field  of  surgical  planning  [fi]  and 
navigation  [4].  In  both  projects  only  the  navigation 
is  enhanced  with  force-feedback.  The  necessary  seg¬ 
mentation  of  the  medical  volumes  is  still  done  with 
traditional  methods. 

The  usage  of  haptic  feedback  to  augment  human 
performance  in  graphical  user  interfaces  is  investi¬ 
gated  from  a  more  general  point  of  view  in  human- 
computer  interaction  research  projects.  In  [I]  it  is 
shown  that  a  mouse  with  force-feedback  can  aid  a 
user  in  targeting  tasks.  The  usage  of  a  haptic  de¬ 
vice  to  lessen  the  information  overload  of  graphical 
displays  is  studied  in  [Uij. 

The  addition  of  force-feedback  to  facilitate  the  un¬ 
derstanding  of  multi-dimensional  data  was  used  in 
[">].  Complex  relationships  between  different  curren¬ 
cies  are  rendered  visually  as  well  as  haptically. 

Generally,  using  multi-modal  interaction  in  volume 
visualization  is  not  a  widely  studied  subject  and  only 
a  few  research  groups  have  begun  to  investigate  the 
possibilities  of  such  an  approach.  A  good  introduc¬ 
tion  to  using  haptics  for  volume  visualization  is  given 


in  [2].  Also  in  [14]  the  benefits  of  haptic  rendering  of 
volume  data  have  been  recognized.  Mainly,  research 
is  carried  out  in  the  field  of  visualization  of  geoscien- 
tific  data  [7,  3,  12,  20]. 

3  Segmentation  Tool 

To  understand  the  effects  of  adding  force-feedback  to 
a  segmentation  program  we  followed  two  different  ap¬ 
proaches.  Firstly,  we  investigated  how  exploration  of 
3D  data  could  be  improved  by  haptic  rendering.  Al¬ 
though  the  main  goal  of  our  research  is  to  improve 
segmentation,  these  initial  studies  gave  us  an  insight 
into  the  way  humans  interact  with  a  multi-modal  in¬ 
terface.  Secondly,  we  examined  a  haptically  assisted 
segmentation  process.  The  reconstruction  of  tubular 
objects  in  the  human  body  served  as  a  case  study 
for  our  system.  The  segmentation  of  these  struc¬ 
tures  is  of  great  medical  interest  and  has  several  ar¬ 
eas  of  application,  for  instance  the  reconstruction  of 
the  nervous  and  the  vascular  system  or  the  bronchial 
tree.  While  haptic  cues  ease  the  three-dimensional 
interaction  and  orientation  in  the  tubular  structures, 
we  present  an  optimal  2D  cross-section  for  visually 
guided  contour  extraction. 

3.1  Haptic  Data  Exploration 

To  give  a  user  a  more  thorough  understanding  of  the 
volume  he  is  interacting  with,  we  mapped  the  gray 
values  of  the  medical  data  to  different  haptic  repre¬ 
sentations.  We  have  to  mention,  that  instead  of  giv¬ 
ing  a  user  a  “realistic”  feel  of  body  tissue  we  only  tried 
to  mediate  information  about  3D  cursor  localization 
and  tissue  densities.  To  this  end  we  combined  hap¬ 
tic  volume  rendering  to  encode  tissue  borders  with 
viscosity  or  vibration  effects  to  display  density  val¬ 
ues.  Due  to  the  limited  continuously  exertable  force 
of  the  haptic  device  and  the  JNDs  of  the  haptic  ef¬ 
fects,  we  chose  to  discriminate  only  between  three 
types  of  densities,  i.e.  bone,  soft  tissue  and  air. 

A  force  field  derived  from  image  gradients  gives  us 
information  about  the  border  location  between  the 
tissue  types  by  resisting  the  motion  from  an  area  with 
small  density  to  an  area,  with  a  larger  one.  After 


thresholding,  we  approximated  the  gradient  vector 
field  of  the  resulting  data  set  by  central  differences. 
Moreover,  to  ensure  the  stability  of  the  computed 
forces,  we  smoothed  the  obtained  gradient  map  with 
a  5x5x5  binomial  filter.  This  force  field  was  precom¬ 
puted  before  the  actual  interaction,  to  ensure  a  fast 
enough  haptic  update.  Because  the  gradient  vectors 
are  located  at  discrete  voxel  positions,  we  have  to  do 
a  tri-linear  interpolation  during  the  interaction  to  ob¬ 
tain  the  continuous  gradient  force  map  Fgrad  needed 
for  stable  haptic  interaction. 

Apart  from  the  gradient  map  we  encoded  the  tissue 
density  itself,  using  vibration  and  viscosity.  The  vi¬ 
bration  effect  is  created  by  sending  a  sinusoidal  force 
with  frequency  and  amplitude  depending  on  gray  val¬ 
ues  to  the  device.  One  interesting  observation  regard¬ 
ing  this  approach  are  the  unintended  audio  signals 
that  were  generated  by  the  vibrating  haptic  device, 
which  gave  a  user  additional  information  about  the 
densities.  The  viscosity  was  computed  according  to 
Stokes  Law,  which  holds  at  low  velocities  in  fields  free 
from  turbulence: 

Fvisc  =  QnrVij, 

where  r  is  the  radius  of  a  sphere  moving  with  velocity 
V'  through  a  medium  with  viscosity  ?/.  The  viscosity 
parameter  was  again  adjusted  according  to  densities. 

Linearly  combining  the  gradient  field  with  the  vis¬ 
cosity  rendering 

F  —  OlFgra(l  -f-  (3Fvisc, 

allowed  a  user  to  do  a  haptically  enhanced  data 
exploration.  Additionally  to  the  force  display,  the 
medical  volume  is  also  volume  rendered  in  stereo  and 
sagittal,  transversal  and  frontal  slices  of  the  data  set 
are  shown.  Furthermore,  the  3D  mouse  can  be  used 
to  place  a  free  oblique  slice  in  the  volume.  During 
some  limited  experiments  with  this  setup  users  re¬ 
ported,  that  tasks  like  tracing  the  tubular  structure 
of  the  colon,  were  much  easier  accomplished  by  rely¬ 
ing  only  on  the  haptics  cues.  This  was  mainly  due  to 
the  complex  task  of  integrating  the  2D  slices  of  the 
volume  into  a  3D  representation,  which  required  a  lot 
of  visual  attention  distracting  from  the  actual  task. 
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The  described  framework  helped  us  to  understand 
more  about  the  psychophysics  of  haptic  interaction 
in  order  to  design  a  system  using  a  force-feedback 
device  in  a  segmentation  process. 

3.2  Multi-modal  Segmentation 

The  initial  step  of  our  multi-modal  approach  is  the 
haptically  assisted  extraction  of  the  centerline  of  a 
tubular  structure.  We  create  forces  that  guide  a  user 
on  a  path  close  to  this  centerline.  In  the  optimal  case 
of  good  data  quality,  the  user  “falls  through”  the  data 
set  guided  along  the  3D  ridge  created  by  the  force- 
feedback.  While  moving  along  the  path,  the  user  sets 
control  points  which  are  used  to  approximate  the  cen¬ 
terline  with  a  spline. 

The  segmentation  of  tubular  structures  can  be  op¬ 
timally  facilitated,  if  all  cross-sections  of  the  object 
are  orthogonal  to  the  centerline.  Therefore,  we  use 
the  interactively  generated  spline  to  determine  the 
appropriate  orientation  of  slice  planes  through  the 
data  volume.  In  a  subsequent  step  an  active  contour 
model  [10]  is  used  to  extract  the  2D  cross-section  of 
our  structure  of  interest  from  these  slices.  The  re¬ 
orientation  of  the  cutting  plane  allows  us  to  make 
a  rough  assumption  about  the  shape  of  the  cross- 
section,  which  should  be  approximately  circular.  Fur¬ 
thermore,  as  we  always  update  the  slice  orientation, 
we  can  use  the  segmentation  of  the  previous  step  as 
a  fairly  good  initialization  for  the  next  cross-section. 
This  can  be  done,  because  the  deviation  between  sub¬ 
sequent  cross-sections  in  a  tubular  structure  is  usu¬ 
ally  small. 

To  create  the  force  field  that  is  guiding  the  user,  we 
adopt  an  approach  similar  to  the  one  used  by  Bartz 
for  enhanced  navigation  [1].  Instead  of  using  the  gray 
values  directly,  we  first  compute  an  Euclidean  dis¬ 
tance  map  of  the  data  set.  S  that:  resulted  from  the 
thresholding  step:  For  each  (x,y.z)  £  S,  the  map 
value 

DM (x,  y,  z)  =  min  d[(x,  y,  z ),  (aq,  yt,  z,)], 

(xi 

is  determined,  where  d  denotes  the  Euclidean  dis¬ 
tance  from  a  voxel  that  is  part  of  the  tubular  struc¬ 
ture  to  a  voxel  of  the  surrounding  tissue.  In  the 


next  step  the  3D  distance  map  is  negated  and  then 
processed  similar  to  the  gradient  rendering  approach. 
Additionally,  to  ensure  stability,  we  apply  a  low  pass 
filter  to  the  computed  forces  over  time. 

In  the  case  of  sufficient  data  quality  this  approach 
works  very  well,  nevertheless,  more  interesting  is  the 
case  of  poor  image  quality,  because  conventional  seg¬ 
mentation  algorithms  do  not  suffice  in  this  situation. 
Here  a.  user  can  be  assisted  by  the  forces  to  bridge 
a  gap  in  the  linear  structure.  We  have  to  mention 
that  a  decrease  of  data  quality  comes  along  with  an 
increased  user  interaction  time  and  effort.  We  intend 
to  overcome  this  shortcoming  with  more  advanced 
gap  closing  mechanisms. 


4  System  Overview 

Our  setup  (Figure  1)  consists  of  two  main  compo¬ 
nents:  an  SGI  Octane  and  a.  haptic  device.  The  Oc¬ 
tane  is  equipped  with  two  R10000  processors  running 
at  195  MIIz  and  512  MB  memory  as  well  as  an  MXI 
graphics  board  with  4  MB  texture  memory,  allowing 
us  to  render  128x128x128  volume  data  sets  in  real¬ 
time. 

We  are  using  a  distributed  architecture  for  our  cur¬ 
rent  prototype  to  ensure  the  necessary  update  rates 
for  the  visual  and  haptic  display.  The  separated 
processes  exchange  data  via  a  shared  memory  with 
semaphore  functionality. 

Only  within  the  last  few  years  suitable  force- 
feedback  devices  became  accessible  for  the  develop¬ 
ment  of  multi-modal  applications  using  touch  interac¬ 
tion.  In  our  project  we  are  using  SensAble  Technolo¬ 
gies’  PHANToM  as  a  sophisticated  3D  force- feedback 
mouse.  The  lack  of  rotational  forces  is  not  a  problem 
for  our  application,  as  we  only  need  point  force  cre¬ 
ation  for  our  current  approach.  Nevertheless,  in  or¬ 
der  to  haptically  encode  the  orientation  of  the  cross- 
sections  we  are  going  to  change  to  a  haptic  device, 
which  also  allows  the  creation  of  rotational  feedback. 


Figure  1:  System  setup 


5  First  Results  and  Future 
work 

In  this  paper  we  presented  a  new  paradigm  for  in¬ 
teractive  segmentation  which  allows  the  handling  of 
medical  data  in  a  natural  way.  Additionally,  we  de¬ 
scribed  how  multi-modal  interaction  can  enhance  the 
process  of  interactive  segmentation.  We  used  this 
approach  to  extract  the  skeleton  of  a  linear  structure 
which  then  served  as  the  basis  for  a  full  segmentation 
of  the  whole  object  of  interest.  In  order  t.o  reduce  the 
interaction  effort  we  have  to  evaluate  more  sophisti¬ 
cated  algorithms  for  gap  closing. 

Moreover,  we  will  study  if  providing  force-feedback 
to  both  hands  can  improve  the  interaction.  Another 
aspect  in  this  direction  is  the  usage  of  different  hap¬ 
tic  devices,  for  instance  a  glove  with  force-feedback 
could  be  used  for  the  non-dominant  hand  to  hold  and 
manipulate  the  data.  Additionally,  we  are  going  to 
investigate  the  possibility  of  including  other  percep¬ 
tual  cues.  The  most  obvious  addition  would  be  the 
auditory  sense. 
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Abstract 

The  deep  tacit  knowledge  of  material,  tool  and 
creative  process,  developed  over  many  years  of 
professional  practice  as  a  sculptor  or  applied  artist  is 
examined  and  outlines  of  an  analysis  of  ‘sensory 
feedback  loops'  present  in  a  specific  making  process  are 
described.  The  significance  and  role  of  the  sense  of  touch 
within  the  development  of  ‘making'  skill  is  discussed  in 
relation  to  ‘embodiment’  (or  transparency)  in  processes 
involving  both  real  and  virtual,  tools  and  materials. 
Conclusions  are  related  to  the  human-computer  interface 
and  specifically  concern  the  role  of  haptic  and  other 
sensory  feedback  in  the  visualisation  and  creation  of 
three-dimensional  virtual  objects.  A  progressive  staged 
model  of  this  is  described  in  relation  to  the 
Artist/Craftsperson(  ‘maker’). 

1.  Introduction  and  background 

The  authors  are  involved  in  detailed  research  surrounding 
the  potential  role  of  haptic  feedback  in  creative  making 
processes.  The  background  and  approach  of  this  research 
arises  from  the  problem  of  reconciling  the  needs  and 
aspirations  of  artists  whose  work  is  grounded  in  the 
making  of  material,  i.e.  non-virtual  three-dimensional 
artworks,  with  the  potential  use  of  computer  visualisation 
systems. 

2.  Methodology  and  analysis 

Reflection  on  the  author’s  glass  sculptural  work  and 
the  unique  techniques  and  processes  that  have  been 
developed  over  25  years  of  professional  practice,  together 
with  an  examination  of  the  visualisation  models  employed 
by  sculptors,  provided  the  basis  of  consideration  of  the 
qualitative  analysis.  This  paper  describes  research  which 
centres  on  an  experimental  view  of  tire  author’s  and  others 


profound  ‘making’  experience.  Issues  of  intentionality 
and  skill  level  within  each  task  would  also  seem  to  be 
significant  factors. 

For  the  purpose  of  the  analysis,  the  task  can  be  broken 
down  into  three  elements,  Human,  Tool,  and  Material. 
The  process,  chosen  by  the  maker,  and  influenced  by  the 
material  and  available  tools  may  be  said  to  act  as  a 
mediator  in  the  task;  all  sensory  feedback  is  evaluated 
within  its  domain.  In  this  basic  scenario,  that  of  unskilled 
beginner  engaged  in  a  non-specific  process,  both  tool  and 
material  are  ‘other’,  neither  are  yet  embodied.  Perception 
is  engaged  by  all  the  senses  and  is  directed  at  both  tool 
and  material  via  process.  Thus  the  task  becomes 
subordinated  to  conscious  and  sub-conscious  filtering  and 
sensory  feedback  evaluation.  In  the  case  of  a  non-specific 
task,  any  or  all  of  the  senses,  touch,  hearing,  vision,  taste 
and  smell  may  be  engaged,  however  the  processes 
themselves  define  which  senses  are  engaged  dependent  on 
the  particular  task. 

In  figure  1 ,  which  refers  specifically  to  a  technique  of 
carving  glass,  the  senses  of  smell  and  taste  are  normally 
irrelevant.  However,  touch,  sound  and  vision  all  convey 
useful  information.  (As  a  novice  with  any  process,  all 
available  sensory  feedback  is  monitored).  The  peculiar 
characteristics  of  the  process  however,  place  vision  in  a 
subordinate  role  to  the  other  two  senses,  since  due  to  the 
refraction  and  reflection  of  the  water  used  as  cutting 
lubricant,  and  the  profusion  of  glass  powder  produced  as 
a  result  of  the  process,  little  can  be  seen. 

Prior  experience  of  similar  processes,  materials  or 
tools  appears  to  speed  up  the  development  of  embodiment 
and  to  make  the  development  of  skill  specifics  more 
rapid.  Figure  2,  examines  the  case  of  a  later  stage  in  skill 
development  where  a  reasonable  change  in  the  constituent 
elements  of  the  feedback  loops. 

This  is  an  interim  development  towards  full  mastery. 
The  tool  has  achieved  embodiment  and  has  become  a 
natural  extension  of  self.  The  amount  of  time  and 
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Figure  1.  Sensory  feedback  pathways  in  glass  carving  by 
rotating  diamond  tool.  (Novice). 


Figure  2.  Sensory  feedback  pathways  in  glass  carving  by 
rotating  diamond  tool.  (Moderate  skill). 

habituation  necessary  to  achieve  this  state  is  variable. 
Perception  in  this  case  appears  to  be  solely  directed 
towards  the  material.  It  is  significant  that  the  sense  of 
touch  has  by  now  become  dominant,  indeed  it  is  difficult 
to  imagine  an  embodiment  that  could  take  place  at  all,  in 
the  context  of  making,  without  a  highly  developed  form  of 
this  sense. 

As  basic  skill  develops  to  ‘mastery’,  a  state  that  takes 
many  years  to  develop,  a  particularly  interesting 


Figure  3.  Sensory  feedback  pathways  in  glass  carving  by 
rotating  diamond  tool.  (Mastery). 


phenomenon  has  been  observed.  Not  only  does  the  tool 
become  embodied  as  described  by  Heidegger  [1],  but  also 
embodiment  grows  to  include  the  material  being  worked. 

In  figure  3,  both  tool  and  material  have  achieved 
embodiment.  The  amount  of  time  and  habituation 
necessary  to  achieve  such  a  state  of  ‘ material 
embodiment’  is  considerably  longer  than  the  previous 
case  that  embodied  only  the  tool.  It  was  seen  that  touch 
was  integral  in  all  parts  of  the  maker’s  sensory  loops,  and 
is  the  only  means  of  response  to  any  information,  no 
matter  by  what  sensory  input  it  arrives. 

Full  embodiment  reduces  response  time  to  a  level  of 
’transparency’.  England  [2]  defines  this  as,  “  ...a  well 
learned  ...  motor  behaviour  which  can  be  performed 
without  conscious  attention.  ”  (p.159.)  Pie  further  states, 
"Movements  faster  than  the  visual  processing  time  (e.g. 
190  ms)  may  therefore  be  under  kinaesthetic  control." 
(p.  161 .)  and  cites  the  studies  of  Carlton,  [3]  and 
Glencross,  [4]  which  seem  to  indicate  that  kinaesthetic 
response  times  may  be  swifter  than  visual  triggering  due 
to  the  differing  physiological  mechanisms  involved. 

The  authors  believe  that  this  analysis  of  skill  development 
indicates  that  if  computer  technology  is  to  be  applied  to 
processes  resulting  in  material  objects,(e.g.  sculpture  or 
industrial  products),  haptic  feedback  is  vital  for  the 
development  of  full  embodiment,  and  that  full 
embodiment  is  necessary  to  achieve  maximal  control  and 
skill  ‘mastery’. 

Performing  a  similar  analysis  of  the  current  standard 
interface  devices  shows  a  picture  where  true  embodiment 
is  simply  not  possible,  since  tool  and  material  both  exist 
in  computer  space.  This  situation  is  somewhat  analogous 
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Figure  4.  Sensory  feedback  pathways  in  current 
standard  human-computer  interface. 

to  attempting  to  re-arrange  the  furniture  in  one’s  house  by 
standing  in  the  garden  poking  a  stick  through  the  window. 

In  figure  4,  touch  feedback  is  restricted  merely  to  the 
physical  interface  device,  mouse  or  keyboard.  No  direct 
contact  is  available  between  the  user  and  that  which  he  / 
she  seeks  to  manipulate.  If  the  material  being  manipulated 
is  solely  abstract  information,  this  might  be  acceptable, 
however  computers  are  increasingly  being  expected  to 
manipulate  informational  representations  of  objects,  real 
or  imagined,  thus  providing  an  alternative  interim 
emphasis.  New  metaphors  must  be  devised  to  elicit  the 
ways  we  personally  model  computer  technology  if  we  are 
to  develop  new  and  more  integrated  ways  of  using  their 
potential.  Coyne,  [5]  identifies  three  basic  metaphors 
implicated  at  the  outset  of  computer  development.  The 
computer  as  “calculating  device”,  the  computer  as  a 
“drawing  tool”  and  the  computer  as  “an  intelligence”.  He 
further  identifies  a  variation  on  the  drawing  metaphor  as: 
"...  to  see  the  computer  as  a  modelling  tool.  Drawing  and 
calculation  conspire  to  construct  a  powerful  metaphor  of 
the  design  model.  A  drawing  is  a  two-dimensional  model. 
Cartesian  geometry  is  used  to  specify  values  for  points, 
lines,  and  planes,  which  are  in  turn  models,  or  perhaps 
metaphors  of  designed  objects.  " 

There  is  however,  a  considerable  experiential  gulf 
between  drawing  a  three  dimensional  model  as  Coyne’s 
metaphor  suggests  and  actually  making  one,  putting  ones 
hands  either  directly  or  via  a  hand  held  tool  onto  a  piece 


of  material,  and  shaping  it  as  one  wills.  The  progressive 
model  outlined  in  figures  1,  2  and  3,  describes  the 
evolutionary  nature  of  material  ‘knowing’.  This 
experience  is  somewhat  reproducible  by  computer 
technology  in  the  domain  of  two  dimensional,  drawing  or 
photo-manipulative  software.  What  the  computer  is 
currently  unable  to  recreate  is  an  authentic  three- 
dimensional  process. 

3.  Conclusions 

Conclusions  are  based  around  the  implicit  hypothesis 
of  positive  tacit  action  in  making  objects.  The  novel 
process  of  analysis  surrounding  a  personal  practice  is 
considered  to  be  a  rich  source  of  qualitative  information. 
This  has  now  been  extended  to  include  other  practitioners 
across  a  range  of  disciplines  so  that  comparisons  may  be 
made. 

•  In  process-based  ‘making’  activities  with  tool  and 
material,  embodiment  (transparency)  develops  with 
skill,  first  the  tool  becoming  embodied,  and  much 
later  the  material  also  becoming  embodied. 

•  This  process  of  embodiment  is  fundamental  in  skill 
development  and  develops  directly  from  the  sense  of 
touch.  It  is  not  primarily  a  cognitively  learned 
process. 

•  Current  computer  interface  technologies  do  not 
address  this  embodiment  of  tool  and  material 
adequately,  rather  there  is  substituted  an  embodiment 
of  the  technology  itself. 

•  The  perceptual  experience  of  a  physical  three- 
dimensional  object  is  markedly  different  to  the 
perception  of  an  iconic  representation  of  that  object 
and  consequently  the  manipulation  of  each  requires 
and  develops  substantially  different  skills. 
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Abstract 

We  present  the  system  for  the  force  -feedback  visu¬ 
alization,  which  uses  hard  realtime  operating  system. 
Software  architecture  uses  two  cooperating  tasks.  The 
first  one  autonomosly  resolves  forces  for  haptic  con¬ 
tact  within  the  small  set  of  triangles  and  it  maintains 
stable  high  update  rate  in  the  range  of  several  kHz. 
The  second  task  runs  in  order  of  magnitude  slower. 
It  selects  triangle  sets  contained  in  limited  subspace 
from  the  virtual  scene  with  the  help  of  Fixed  Direction 
Hulls. 

Keyword:  PHANToM,  haptics,  RT-Linux,  virtual 
environment,  fixed  direction  hull,  force  feedback 

1  Introduction 

Tb  get  the  sense  of  touch  we  add  haptical  rendering 
to  the  graphical  visualization.  For  haptical  visualiza¬ 
tion  we  use  force-feedback  device  PHANToM.  For  the 
best  experience  graphical  and  haptical  systems  have  to 
run  fast  and  fluently.  Processing  complicated  analytic 
surfaces  like  NURBSfl]  is  still  highly  computation  in¬ 
tensive  task.  As  we  use  OpenGL  for  graphical  visu¬ 
alization  it  is  natural  to  use  uniform  representation 
based  on  triangle  sets  and  meshes.  Hence  we  trian¬ 
gulate  all  complex  models  including  analytic  surfaces 
before  we  visualize  them. 

In  this  article  we  present  our  approach  for  haptical 
visualization  of  large  triangle  sets  on  PHANToM  de¬ 
vice  using  hard  realtime  operating  system  RT-Linux1. 

With  this  system  we  try  to  avoid  the  computation 
of  complicated  surfaces  in  the  realtime  as  they  could 
be  precomputed  into  triangle  meshes  without  the  no¬ 
table  loss  of  quality. 

We  have  decided  when  we  have  already  triangulated 
objects  for  3D  rendering,  it  would  be  good  to  use  the 
similar  representation  (preferably  the  same  one)  also 

1  http://www.rtlinux.org 


for  haptical  rendering.  Simple  geometrical  objects  like 
spheres,  toroids  and  various  other  shapes,  which  could 
be  described  by  algebraic  equations,  are  more  precisely 
haptically  rendered  using  the  analytic  approach.  How¬ 
ever  in  the  real  life  scenes  it  is  hard  to  find  such  ob¬ 
jects  and  we  usually  work  with  huge  triangle  meshes 
from  3D  modelling  tools.  Moreover  these  triangle  sets 
commonly  contain  problematic  areas  and  its  not  even 
specified  what  it  means  to  be  inside  or  outside  of  an 
object. 

Some  researchers  have  already  presented  their  al¬ 
gorithms  for  haptical  rendering  of  triangles,  but  these 
are  usually  not  prepared  to  work  reliable  with  update 
frequencies  in  the  range  of  a  few  kHz. 

In  this  article  we  also  mention  some  preprocess¬ 
ing  steps  we  use  to  minimalize  problems  with  more 
or  less  incorrect  triangle  models.  These  are  usually 
being  generated  by  various  3D  modelling  tools. 

2  Previous  Work 

To  achieve  hard  surface  with  haptical  rendering  we 
need  high  update  rates  (at  least  1kHz)  and  fast  meth¬ 
ods  for  evaluation  of  reacting  forces.  Contact  points 
necessary  to  compute  forces  are  usually  found  with 
the  help  of  collision  detection  systems.  That  means 
we  have  to  use  very  efficient  and  fast  algorithms  here. 

A  good  overview  of  traditionally  used  methods  in¬ 
cluding  oriented  bounded  boxes  (OBB)  and  axis  align  ¬ 
ed  bounding  boxes  (AABB)  and  bounding  volume  hi¬ 
erarchies  (B VH)  can  be  found  in  [2] .  For  the  haptical 
rendering  few  special  collision  detection  systems  have 
been  developed:  H-Collide,  I-Collide  see:  [3].  These 
systems  are  usually  based  on  OBB  and  they  focus  on 
detection  of  one  surface  contact  point. 

For  the  system  with  two  levels  of  computation  hier¬ 
archy  similar  approach  could  be  found  in  ArmLib[4]. 
This  project  uses  an  equation  of  a  single  plane  which  is 
maintained  by  the  fast  internal  loop  while  the  second 
task  updates  this  plane  with  much  slower  rate. 
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3  Method  Overview 

Our  goal  was  to  develop  the  method  which  could 
be  used  generally  to  haptically  visualize  the  same  data 
sets  as  are  used  for  graphical  visualization. 

The  algorithm  we  present  here,  is  divided  into  two 
parts  running  in  two  different  environments.  The  first 
part  is  running  in  the  normal  user  space  mode  and  its 
purpose  is  to  select  a  small  subset  of  triangles  from 
the  large  virtual  scene.  The  second  part  is  running 
in  the  realtime  environment  and  has  limited  capabil¬ 
ities.  The  computation  time  is  bounded  and  also  the 
memory  access  is  limited  in  this  environment.  So  the 
algorithm  has  to  be  simple  and  fast.  This  task  works 
with  some  small  neighbourhood  for  a  given  point  and 
surrounding  triangles  proportionally  influence  the  re¬ 
sulting  force. 

Using  two  separate  algorithms  where  one  is  used  to 
pick  small  surrounding  virtual  environment  from  the 
large  one  and  the  second  task  is  resolving  haptic  feed¬ 
back  within  this  smaller  subset,  might  be  viewed  as  a 
special  case  of  3D  traversal  exploiting  frame-to-frame 
coherence.  For  the  hierarchical  representation  of  the 
scene  we  use  BVH  of  FDH  which  will  be  briefly  de¬ 
scribed  later.  We  use  two  levels  of  BVHs.  The  first 
level  contains  all  objects  and  it  is  used  to  detect  col¬ 
lision  between  them.  The  second  one  is  used  to  select 
few  triangles  from  the  colliding  object. 

4  Algorithm 

In  this  section  we  describe  how  the  system  pro¬ 
cesses  triangle  sets.  We  present  new  algorithm  used  for 
calculation  of  reacting  forces  which  also  does  smooth 
shading  of  normal  vectors. 

4.1  Preprocessing 

For  OpenGL  rendering  number  of  various  optimal- 
ization  have  been  developed.  With  todays  graphics 
cards  we  do  not  have  to  care  too  much  if  we  will  draw 
several  hundred  triangles  more  or  less.  However,  with 
the  haptic  rendering  the  situation  is  different.  We  need 
to  eliminate  as  much  triangles  as  we  can  before  pro¬ 
cessing  them  in  the  force  loop. 

Usually  it  is  better  when  the  scene  consists  of  large 
triangles.  But  at  certain  size  large  triangles  cause  the 
problem  as  they  need  too  large  bounding  volumes.  To 
create  more  compact  bounding  volumes  we  have  to 
split  larger  triangles  into  the  smaller  ones.  Small  tri¬ 
angles  can  be  more  easily  sorted  in  a  balanced  BVH. 

On  the  other  hand  too  small  triangles  which  might 
appear  in  some  detailed  models  have  to  be  reduced 
otherwise  the  selected  environment  would  be  too  small 


and  user  would  shortly  escape  from  ’precached’  tri¬ 
angle  set.  For  this  purpose  right  now  we  preprocess 
triangle  sets  with  surface  simplification  algorithm[5]. 

Preprocessing  algorithm  consists  of  following  steps: 

•  create  the  list  of  triangles  and  find  common  edges 

•  with  respect  to  sharp  edges  fix  or  fill  missing 
vertex  normals 

•  check  if  adjanced  triangles  have  the  same  nor¬ 
mal  vectors  in  common  vertices;  in  the  case  nor¬ 
mals  differ  add  two  fake  triangles  (fig.l)  with 
only  two  different  points  which  will  assure  con¬ 
tinuous  change  of  the  normal  vector. 


Figure  1:  Adding  two  fake  empty  triangles. 

•  split  large  triangles 

•  final  validation  of  normal  vectors 

4.2  Computation 


Figure  2:  Force  calculation. 

Algorithm  1.  for  the  small  triangle  set  works  as 
follows:  For  each  triangle  (fig.2)  we  check  first  if  it  has 
the  proper  orientation  and  distance  with  respect  to  the 
current  position  P  of  PHANToM.  Then  we  continue 
with  the  following  step.  For  each  edge  of  the  triangle 
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Algorithm  1  Force  calculation 
force  =  (0,0,0) 
max  force  —  0 
for  all  T  €  object  do 
d  ■$=  distance(P ,  T) 
if  d  >  0  and  d  <  RADIUS  then 
for  i  =  0  to  3  do 
B  nearestpoint(edge(T,i),  P) 
m  <=  interpolation^ ,i  ,B) 

R  <=  plane(B,ni,edge(T,  i)) 
di  <=  distance(P ,  R) 
end  for 

if  do  >  0  and  d\  >  0  and  d2  >  0  then 


no  <=  <=  ai, n2  4= 


n  4=  no  +  ni  + 


D  4=  inter section(n,T) 
if  \D$\  <  RADIUS  then 
s  4=  f(DP) 
force  4=  force  +  s 
max  force  4=  max(max  force,  |s|) 
end  if 
end  if 
end  if 
end  for 


force  -  j/orce  *  max  force 


We  resolve  all  VCPs  and  we  remove  those  which 
are  actually  hidden  bellow  some  other  triangles  (fig.3). 
This  could  be  easily  achieved  just  by  checking  if  the 
line  between  VCP  and  real  position  P  is  not  intersect¬ 
ing  another  triangle  with  VCP.  Finally  we  combine  all 
forces  together.  Resulting  force  is  actually  reasonably 
good  approximation  which  is  a  compromise  between 
the  speed  of  calculation  and  the  precision  of  the  re¬ 
sulting  force. 

Now  we  have  smooth  normal  vectors  without  any 
bouncing  change  in  the  direction,  but  we  still  do  not 
know  whether  user  actually  penetrates  the  surface.  To 
solve  this  problem  we  use  the  following  procedure: 


we  find  the  closest  point  and  by  linear  interpolation 
we  compute  the  normal  vector  at  this  point.  Normal 
vector  and  edge  define  plane  R.  If  the  point  P  is  bellow 
this  plane  we  stop  processing  this  triangle.  Now  we 
have  three  normals  and  distances  from  planes.  Each 
normal  is  divided  by  its  distance  from  the  plane  R 
to  get  proportional  force.  Summing  all  three  of  them 
we  get  the  total  force  which  is  pushing  the  user  away 
from  this  triangle.  Now  we  could  also  compute  virtual 
contact  point  (VCP)  D  as  the  intersection  of  the  line, 
which  goes  through  the  point  P  in  direction  of  the 
resulting  normal,  with  the  triangle’s  plane. 


Figure  3:  Overlapping  triangles. 


Figure  4:  Penetration  of  the  surface. 

If  during  our  calculation  there  is  no  intersection  on 
path  between  two  consecutive  points  e.g.  Pq  and  Pi 
(fig. 4)  with  any  of  processed  triangles  then  the  user  is 
not  inside  the  object. 

If  such  point  is  found  then  we  use  the  VCP  (Vo)  for 
the  force  calculation.  In  the  following  iteration  we  add 
difference  vector  (P1P2)  to  Vo  and  we  try  to  find  new 
intersection  points.  We  take  into  account  only  those 
triangles  where  we  detect  the  movement  from  inside 
to  outside  half  space.  In  the  case  of  more  than  one 
intersection  we  pick  the  closest  point  (To)  and  use  it 
for  calculation  of  the  new  VCP  (Vj).  If  there  is  no 
such  point  we  use  current  VCP. 

5  Fixed  Direction  Hulls 

To  select  a  small  amount  of  triangles  from  a  sub¬ 
space  we  need  to  use  very  fast  method.  For  static 
scenes  we  have  achieved  good  results  with  Fixed  Di¬ 
rection  Hulls  also  known  as  k-Dops  (Discrete  Oriented 
Polytops)  see  fig.5.  Details  can  be  found  in  [6], [7]. 

FDHs  have  almost  the  same  processing  speed  as 
AABBs,  but  they  bound  objects  more  tightly,  just 
like  OBBs.  An  advantage  of  FDH  is  lower  query  time. 
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FDH14 

Figure  5:  FDH14  and  2-D  example  (FDH8). 

On  the  other  hand  their  disadvantage  is  that,  there 
is  no  simple  solution  for  rebuilding  FDII  after  rota¬ 
tional  transformations.  Hence  we  prefer  this  method 
for  static  scenes  where  only  few  objects  move  (user’s 
hand,  in  our  case).  We  use  BVHs  which  are  composed 
from  FDH14.  One  FDHm  is  internally  represented 
by  14  floating  point  numbers  which  define  14  cutting 
planes  with  given  normals.  The  first  fi  planes  are  the 
same  as  in  AABB.  Most  of  the  computations  when 
testing  collisions  in  rarely  populated  scenes,  have  the 
same  averrige  complexity  as  when  using  AABB. 

6  Implementation  Details 

For  implementation  of  this  system  we  developed 
driver  for  hard  real  time  OS  RT-Linux.  We  are  able 
to  test  wide  spectrum  of  update  rates.  Hence  in  this 
area  we  are  exceeding  the  possibilities  of  the  original 
software  supplied  with  PHANToM  device.  While  the 
standard  implementation  for  Window  NT  operates  on 
1kHz  update  rate  we  are  able  to  use  rates  up  to  10kHz2 
with  the  possibility  to  change  this  speed  in  runtime. 

For  our  experiments  we  use  dual  processor  system 
with  two  470MHz  Celerons  running  RT-Linux  2.4.5 
and  Matrox  G400  3D  acceleration  graphics  card. 

With  our  demonstration  application  we  use  polyg¬ 
onal  models  consisting  from  around  20000®  triangles. 
We  achieve  FDH  query  time  within  the  range  of  10ms 
and  less  (typically  around  3ms)  while  the  hard  real¬ 
time  loops  runs  with  64  selected  triangles  and  the  sys¬ 
tem  still  show  around  50%  idle  time  with  3kHz  update 
rate. 

7  Conclusions  and  Future  Work 

Currently  we  can  process  only  static  scenes,  but  we 
prepare  implementation  of  dynamic  algorithm  which 
will  allow  us  to  update  triangle  sets  and  to  do  a  smooth 
transition  between  two  sets. 

According  to  our  experiments  [10]  this  appears  to  he  rather 
a  hardware  limit  for  this  device  with  PCI  card. 

3so  they  could  be  displayed  with  acceptable  frame  rate  on 
the  graphics  card 


Availability 

The.  latest  informations  about  our  software,  which 
is  available  under  GPL,  can  be  found  on  our  web  page: 
http://decibel.fi.muni.cz/phantom/ 
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Abstract— Current  computational  simulation  are  ca¬ 
pable  of  producing  enromous  amounts  of  data.  Com¬ 
plete  understanding  of  their  features  presents  a  chal¬ 
lenge  even  if  very  sophisticated  visualization  tech¬ 
niques  are  deployed.  Computational  analysis  of  con¬ 
formational  behaviour  of  biologically  active  com¬ 
pounds  represents  such  simulation.  We  investigate 
methods  how  haptic  rendering  may  contribute  to  bet¬ 
ter  and  faster  understanding  of  the  simulation  result- 
s.  This  paper  presents  current  progress  in  our  re¬ 
search. 

I.  Molecular  Flexibility  and  Conformational 
Behaviour 

Many  molecules  exhibit  an  important  chemical 
property  — flexibility.  Biological  activity  of  large 
molecules  is  directly  related  to  their  flexibility.  The 
flexibility  can  be  described  in  terms  of  conforma¬ 
tional  behaviour  of  the  molecule.  Undergoing  the 
behaviour,  the  molecule  changes  its  shape  ( config¬ 
uration )  only,  no  chemical  bonds  are  either  creat¬ 
ed  or  broken,  as  well  as  the  absolute  configura¬ 
tion  on  atoms  (or  other  chiral  centres)  does  not 
change.  Roughly  speaking,  internal  potential  en¬ 
ergy  of  the  molecule  is  a  function  of  the  configu¬ 
ration,  therefore,  in  general,  not  all  the  configura¬ 
tions  are  favoured  equally.  By  conformations  we 
mean  local  potential  energy  minima,  i.  e.  the  con¬ 
figurations  that  are  more  stable  than  the  others. 
Then  the  conformational  behaviour  is  the  process 
of  traversing  among  conformations  via  transition 
states. 

Various  computational  methods  of  discovering 
conformations  of  a  molecule  and  describing  its 
conformational  behaviour  were  proposed.  It  is  be¬ 
yond  the  scope  and  purpose  of  this  article  to  pro¬ 
vide  even  an  overview.  In  our  application  we  use  re¬ 
sults  of  the  Cicada  family  of  programs  performing 
heuristic  search  of  conformational  space  [1],  They 
produce  a  graph  of  interconnected  conformation- 
s  and  transition  states,  i.  e.  saddle  points  in  terms 
of  potential  energy.  Two  edges  of  the  graph  that 


share  a  common  transition  state  represent  a  traver¬ 
sal  path  of  minimal  energy  barrier  between  a  pair 
of  conformations. 

II.  Haptic  Model 

In  this  section  we  focus  on  the  description  of  the 
model  being  deployed  in  our  application  and  its 
mathematical  aspects.  Various  implementation  is¬ 
sues  like  program  structure,  performing  exhaustive 
computation  off-line  etc.  are  discussed  in  detail  in 
Sect.  in. 

A.  Virtual  Energy  and  Haptic  Steering 

We  build  the  model  on  the  principle  of  haptic  s- 
teering,  i.  e.  the  user  is  provided  with  a  virtual  tool 
bound  to  the  haptic  device.  With  the  tool  she  is 
able  to  interact  with  the  virtual  model.  The  inter¬ 
action  enforces  changes  in  the  virtual  world  that 
incorporate  changes  of  the  potential  energy  of  the 
model.  According  to  well-known  physical  laws  the 
work,  i.  e.  the  energy  produced  or  consumed  by 
a  system,  can  be  computed  by  integrating  force  in¬ 
teraction  along  a  path.  Hence,  the  force  feedback 
can  be  computed  as  a  spatial  gradient  of  the  ener¬ 
gy- 

The  model,  despite  depicting  a  micro-world 
scene,  should  behave  intuitively  in  terms  of  macro¬ 
world  physical  laws  in  order  to  be  convincing.  In 
our  everyday  life  we  witness  consequences  of  the 
second  thermodynamic  law  — spontaneous  energy 
minimization.  Any  natural  system  without  an  ex¬ 
ternal  intervention  does  not  stay  in  an  instable  s- 
tate,  it  follows  its  potential  energy  gradient  until 
reaching  a  local  minimum.  Therefore,  in  order  to 
achieve  an  intuitive,  convincing  behaviour  of  the 
virtual  model,  we  have  to  simulate  the  energy  min¬ 
imization.  For  this  purpose  we  introduce  a  concept 
of  virtual  energy  that  covers  both  the  properties  of 
the  model  we  intend  to  render  as  well  as  physical 
properties  that  resemble  a  real-world  object. 
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Fig.  1.  Example  conformation  and  four  transition  states  of  the  alanine  amino  acid.  The  numbers  show  energy 
barriers  (in  kcal/mol) 


B.  Molecular  Model  Setup 

The  current  implementation  restricts  the  model 
to  a  chosen  conformation  and  its  surroundings,  i.  e. 
it  spans  upto  the  set  of  enclosing  transition  states. 
An  example  is  shown  in  Fig.  1.  We  are  able  to  com¬ 
pute  an  arbitrary  point  on  a  path  between  the  con¬ 
formation  and  a  transition  state.  However,  for  the 
purpose  of  haptic  interface,  we  have  to  cover  the 
intermediate  configuration  as  well.  It  is  general¬ 
ly  acceped  as  an  affordable  simplification  that  the 
conformational  paths  can  be  expressed  in  terms  of 
torsions,  i.e.  twisting  the  molecule  along  a  par¬ 
ticular  bond  (or  more  bonds  simultaneously)  — 
this  can  be  seen  in  Fig.  1. 

Therefore,  given  position  on  the  paths,  we  are 
able  to  construct  a  merged  configuration1  by  aver- 

xIt  has  to  be  emphasized  that  this  merge  is  artificial,  with 
no  direct  background  in  chemistry.  Its  purpose  is  building 
a  continuous  state  space  that  has  to  be  present  in  a  haptic 
model.  However,  we  are  really  interested  in  the  precomputed 
conformational  paths  only. 


aging  those  twist  angles. 

The  user  is  presented  a  van  der  Waals  molecular 
surface  (atoms  are  represented  by  spheres,  over¬ 
lapping  in  general,  see  Fig.  2)  both  in  the  haptic  and 
graphical  interface.  A  3 -DOF  haptic  device  is  at¬ 
tached  to  a  virtual  probe ,  an  approximately  water¬ 
sized  sphere. 

C.  Energy  and  Force  Computation 

Virtual  energy  of  this  model  depends  on  the 
probe  position  and  the  actual  configuration  of  the 
molecule,  and  obeys  the  following  rules: 

.  Penetration  of  the  surface  by  the  probe  is 
strongly  penalized  but  not  completely  forbid¬ 
den.  The  interaction  of  the  probe  with  the 
surface  is  the  only  way  of  enforcing  a  shape 
change,  therefore  the  configuration  change 
should  be  favoured.  On  the  other  hand,  the 
amount  of  penetration  is  a  direct  and  proven 
clue  to  the  computation  of  force  feedback, 
e.g.  [2],  [3],  [4]. 
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Fig.  2.  Van  der  Waals  surface  of  the  alanine  amino  acid 


•  The  configuration  paths  between  the  central 
conformation  and  the  transition  states  are 
supposed  to  go  through  “valleys”  of  the  true 
potential  energy.  Therefore  the  precomputed 
paths  are  favoured  in  comparison  to  other  con¬ 
figurations.  In  addition,  according  to  the  input 
data  structure  we  are  given  direct  information 
on  the  potential  energy  on  those  paths  only. 

•  Auxiliary  terms  that  selectively  penalize  shift 
of  atoms  from  their  initial  position  (see  [5])  are 
included.  As  a  consequence  shape  modifica¬ 
tions  close  to  the  probe  are  easy  to  do  while 
the  molecule  is  still  fixed  in  space. 

In  each  iteration  of  the  interface  driving  loop  we  are 
given  current  coordinates  of  the  probe.  For  the  pur¬ 
pose  of  energy  minimization,  we  treat  it  as  a  con¬ 
figuration  constraint  on  the  model. 

The  degrees  of  freedom  of  the  model  are  ex¬ 
pressed  in  terms  of  its  free  variables.  Basically, 
there  are  two  classes  of  them:  positions  on  the 
precomputed  conformation-transition  state  path- 
s  which  represent  the  actual  shape  of  the  molec¬ 
ular  model,  and  overall  shift  and  rotation  of  the 
molecule  defining  an  absolute  position  of  the  mod¬ 
el  in  3D  space. 

Given  values  of  the  free  variables  the  shape  of 
the  molecule  can  be  computed.  The  virtual  ener¬ 
gy  follows  the  results  of  chemical  calculation  in 
case  of  paths  between  the  central  conformation 
and  transition  states,  and  is  raised  artificially  (but 
coincidently  with  chemical  expectations)  to  form 
barriers  in  areas  between  those  paths.  Then  the 
molecule  is  located  in  3D  according  to  the  sec¬ 
ond  class  variables.  Now  the  probe-penetration  and 
position-stabilizing  terms  can  be  computed  as  well. 


The  minimization  starts  from  the  last  known  s- 
tate,  i.  e.  free  variable  values.  The  energy  minimiza¬ 
tion  outputs  a  favoured  shape  and  position.  Those 
are  shown  directly  by  the  graphical  interface.  As 
the  energy  contribution  of  the  probe  interaction 
with  the  surface  is  proportional  to  the  resulting 
amount  of  penetration  only,  the  force  interaction 
can  be  derived  from  it  directly. 

From  the  user’s  point  of  view  the  molecular  mod¬ 
el  resembles  a  slippery  flexible  object.  It  is  possible 
to  modify  its  shape  by  pushing  the  probe  against  it. 
Certain  shape  changes  (those  corresponding  to  the 
energy  “valleys”)  are  strongly  prefered  to  others. 

III.  Implementation  Issues 

Let's  focus  on  the  computational  requirements 
of  the  above  model  now.  The  computations  can  be 
classified  as  follow's: 

.  shape  and  position  from  the  free  variables, 
(hence  the  force  interaction)  —  fast  enough  to 
be  computed  inside  the  haptic  interface  loop 
(i.e.,  at  1-5  kHz  rate), 

.  energy  minimization  —  typically  several  hun¬ 
dreds  shape  computations,  cannot  be  done  in¬ 
side  the  haptic  loop. 

Therefore  we  perform  all  the  energy  minimization- 
s  offline,  on  a  regular  grid  of  the  probe  positions. 
For  the  PHANToM  device  we  use  200x260x85  grid 
covering  most  of  the  device  workspace.  Moreover, 
different  states  (i.  e.,  free  variables  vector)  can  be 
reached  if  the  same  probe  position  was  approached 
from  different  previous  states.  Therefore  we  have 
to  introduce  another  dimension  (a  level)  to  the  grid, 
and  keep  track  of  transitions  between  the  levels  as 
well.  Fortunately,  level  transitions  are  quite  rare, 
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the  grid  size  is  usually  increased  by  factor  less 
than  2.  With  the  given  resolution  the  grid  size  is 
less  than  10  M  points.  We  do  not  need  high  reso¬ 
lution  of  the  free  variables  (1  byte  is  enough),  and 
a  typical  conformation  of  molecules  we  work  with 
can  be  described  with  upto  20  variables.  As  the  s- 
tate  space  is  continuous  we  can  traverse  the  grid 
in  6  orthogonal  directions  only,  keeping  the  infor¬ 
mation  on  level  transition  in  4  bils.  Therefore  we 
end  up  with  maximal  grid  file  size  250MB,  which  is 
generally  affordable. 

At  run  time,  we  compute  trilincar  interpolation 
of  free  variables’  values  according  to  the  probe  po¬ 
sition  in  the  grid.  The  force  interaction  is  comput¬ 
ed  from  the  interpolated  values. 

The  prototype  application  consists  of  three  pro¬ 
grams:  offline  grid  computation,  force  feedback 
device  (PHANToM)  driving,  and  visualization.  The 
current  implementation  of  the  energy  minimiza¬ 
tion  uses  the  NAG  numerical  library,  restricting  us 
to  the  SGI  platform  by  our  license  terms,  but  this  is 
not  a  general  concern.  The  PHANToM  driving  runs 
on  a  RT-Linux  machine  using  a  driver  developed  at 
our  site2.  The  driver  allows  high  refresh  rates  (over 
10  kHz)  of  the  haptic  device  as  well  as  loading  crit¬ 
ical  parts  of  the  code  into  the  Linux  kernel.  In  this 
way  a  guaranteed  response  time  of  the  device  driv¬ 
ing  is  achieved. 

We  use  VMD  — a  publicly  available  chemical  vi¬ 
sualization  tool3  for  the  visual  rendering  of  the 
scene.  VMD  is  a  complex  tool,  besides  differen- 
t  visual  representations  of  the  molecular  model  it 
allows  real-time  measure  of  geometry  properties 
(distances,  angles,  ...)  etc.  The  system  defines 
a  network  communication  protocol  (originally  de¬ 
signed  for  interaction  with  the  NAMD  molecular  dy¬ 
namics  program)  that  is  used  to  feed  it  with  geom¬ 
etry  data  from  the  force  feedback  driving. 

IV.  Conclusions  and  Future  Work 

We  build  a  haptic  model  of  a  molecule  and  its  be¬ 
haviour  in  surroundings  of  a  chosen  conformation. 
The  model  is  realistic  in  the  means  that  it  strictly 
follows  results  of  chemical  calculation  in  “valleys” 
of  potential  energy,  i.  e.  the  areas  that  are  inter¬ 
esting  to  the  user.  The  information  on  energy  of 
the  current  state  of  the  model  is  delivered  via  the 
force  feedback.  This  proves  to  be  a  natural  and  in¬ 

2http: //decibel .fi .muni .cz/phantom/ 

3http://www. ks.uiuc.edu/Research/vmd/ 


tuitive  representation  of  the  quantity.  As  the  mod¬ 
el  is  highly  interactive  it  provides  an  immediate  in¬ 
sight  on  the  behaviour. 

The  work  presented  in  this  article  is  a  continua¬ 
tion  of  recent  research  [6],  [5].  There  are  two  prin¬ 
cipal  improvements  in  the  current  implementation: 

•  The  model  follows  real  precomputed  confor¬ 
mational  paths  instead  of  relying  on  bond  flex¬ 
ibility. 

.  Most  of  the  exhaustive  calculation  is  per¬ 
formed  offline.  Therefore  the  interactivity  of 
the  model  improved  significantly. 

In  the  future  we  are  going  to  focus  on  two  exten¬ 
sions  of  the  current  model.  The  first  one  is  over¬ 
coming  the  restriction  of  the  user’s  workspace  to 
a  single  conformation.  Once  the  user  delivers  suf¬ 
ficient  force  in  a  proper  direction  to  overcome  an 
energy  barrier  of  a  particular  conformational  path, 
the  molecular  model  should  “flip”  to  the  other  con¬ 
formation.  We  believe  this  is  a  technical  problem 
only  as  the  energy  function  as  defined  in  this  ar¬ 
ticle  overlaps  near  transitions  states  not  regarding 
from  where  the  state  is  approached. 

The  second  possible  extension  we  can  see  is 
more  challenging.  In  order  to  explore  more  com¬ 
plex  molecules  in  their  full  features  more  haptic  de¬ 
vices  (maybe  more  DOF’s  per  device)  are  required. 
However,  straightforward  increasing  the  number  of 
DOF’s  increases  the  dimension  of  the  precomputed 
grid  file  which  is  not  affordable  generally.  There¬ 
fore  we  shall  have  to  look  for  a  smarter  solution. 
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Abstract 

This  paper  addresses  key  perceptual  and  psychophysical 
issues  concerned  with  the  design  of  a  multiple  contact 
point  haptic  device.  The  majority  of  commercially 
available  haptic  devices  operate  based  on  a  principle  of 
point  interaction.  Thus,  contact  between  the  operator  and 
the  simulated  virtual  environment  occurs  only  at  an 
arbitrary  single  point,  typically  the  tip  of  a  stylus  or 
thimble  used  for  interaction.  However,  this  can  severely 
impair,  or  even  exclude,  perception  of  object  material 
properties.  It  is  shown  that  a  haptic  interface  mechanism 
incorporating  several  points  of  contact,  corresponding  to 
the  tips  of  different  fingers,  for  example,  may  facilitate 
more  optimal  acquisition  of  information  regarding  object 
properties.  An  example  of  how  several  grounded  devices 
may  be  positioned  to  combine  their  workspace  is 
presented,  achieving  a  workspace  of  roughly  20x30x15 
cm. 

1.  Introduction 

When  haptically  exploring  an  object,  it  has  been  shown 
that  the  explorer  unconsciously  adopts  a  series  of 
stereotypical  hand  movements,  in  order  to  extract  the 
desired  information  in  an  optimal  fashion  [4].  Each 
exploratory  procedure  (EP)  is  associated  with  a  particular 
object  dimension,  for  which  it  is  the  optimal  and  preferred 
method  for  determining  the  property  under  unconstrained 
haptic  exploration.  Experimental  results  clearly  indicated 
that  in  a  free  exploration  t  ask,  hand  movement  was 
related  to  desired  knowledge.  EPs  adopted  during  free 
exploration  are  usually  optimal,  if  not  necessary,  for 
extracting  the  information  required. 

When  using  a  haptic  interface  in  order  to  explore  a 
simulated  or  remote  environment,  it  is  inevitable,  given 
the  mechanical  constraints  imposed  by  the  design  of  such 
devices,  that  performance  of  EPs  is  impaired,  and  in  some 
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cases  totally  occluded.  Thus,  the  user  is  forced  to  adopt 
sub-optimal  exploration  strategies  in  order  to  extract 
properties  of  objects.  This  problem  is  particularly 
accentuated  for  those  devices  that  operate  on  a  principal 
of  point  interaction  via  a  thimble  or  probe. 

Given  such  constraints,  table  1  summarises  the  principle 
EPs  and  their  availability  during  point  interaction.  Hence, 
lateral  motion,  pressure  and  contour  following  are 
unaffected  by  the  point  interaction  method,  though  the 
cues  they  elicit  will  differ  from  those  obtained  via  direct 
contact.  Unsupported  holding,  given  the  formal  definition 
above,  is  not  possible.  However,  it  is  possible  to  simulate 
weight  sensations  by  allowing  virtual  objects  to  be 
"stuck"  to  the  distal  point  of  the  probe.  In  theory,  static 
contact  is  also  possible,  though  temperature  actuators 
have  yet  to  find  widespread  application  in  the  field. 
Finally,  enclosure  is  impossible  to  perform  via  point 
interaction,  as  at  least  two  points  are  required  to  begin 
attempting  to  mould  to  an  object’s  global  shape. 

It  has  been  shown  that  material  properties  of  objects  (for 
example,  stiffness  and  texture)  obtain  a  higher  salience 
under  purely  haptic  exploration,  whereas  structural  cues 
are  more  easily  obtainable  and,  hence,  more  salient  for 
discrimination  under  visual  exploration  [2], 

However,  results  suggest  that  perception  of  material 
properties,  such  as  texture,  is  impaired  due  to  the 
bandwidth  of  currently  available  haptic  interface 
technology  [7,  11,  12].  Thus,  operators  of  haptic 
interfaces  are  forced  to  adopt  a  "visually  mediated" 
approach  to  object  discrimination,  that  is  to  say,  that 
performance  must  rely  on  visual  stimuli  provided  by  a 
suitable  display,  or  that  the  user  must  gather  information 
regarding  object  properties  that  are  more  readily  encoded 
by  vision,  such  as  size  and  shape. 

Current  haptic  interface  technology,  however,  provides  no 
facilities  for  the  enclosure  EP,  which  is  optimal  for  global 
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shape  and  volume.  Point  interaction  devices  are  also 
deleterious  to  the  execution  of  contour  following,  which 
becomes  time  consuming  and  difficult  due  to  the  greater 
memory  demands  for  temporal  integration  of  cues.  It  was 


shown  that  exploration  via  a  single  contact  point  increases 
response  times  in  tasks  requiring  comprehension  of 
geometric  data  [5].  A  similar  effect  was  observed  between 
full  handed  and  single  fingered  exploration  of  objects[3]. 


Exploratory  Procedure 

Description 

Availability  under  point 
interaction 

Lateral  Motion 

Movement  back  and  forth  between 
skin  and  object  surface. 

Available,  though  cues  will  be 
temporally  varying  (vibration) 
instead  of  spatially  varying. 

Pressure 

Applying  normal  force. 

Available 

Static  contact 

Resting  passively  without  moulding. 

Available,  though  no 
temperature  or  distributed  force 
cues  are  available  through  probe. 

Unsupported  Flolding 

Object  is  lifted  and  maintained. 

Available,  by  attaching 
simulated  object  to  distal  point 
of  probe. 

Enclosure 

Hand  maintains  simultaneous  contact 
with  as  much  of  object  as  possible. 

Unavailable  in  absence  of 
multiple  contact  points. 

Contour  Following. 

Smooth,  non-repefitive  movement 
with  a  contour  of  the  object. 

Available.  Difficult  to  execute 
due  to  small  contact  area. 

Table  1:  Availability  ofEPs  with  point  interaction. 


2.  Multiple  Contact  Point  System 

Given  the  limitations  of  currently  available  point 
interaction  devices,  a  system  is  proposed  in  which  several 
commercially  available  devices  (the  PHANToM 
(www.sensable.com))  are  combined  in  order  to  facilitate 
multiple  fingertip  interactions.  It  is  envisaged  that  the 
system  will  allow  grasping  and  enclosure  procedures 
previously  unavailable  with  haptic  devices,  and  will  also 
potentially  improve  efficiency  in  extraction  of  object 
properties.  For  example,  increasing  the  number  of  fingers 
stimulated  by  similar  tactile  sensations  increases  the 
perceived  vibration  magnitude  by  a  rule  of  diminishing 
returns;  thus,  by  using  multiple  stimulation  points  it  may 
be  possible  to  increase  the  perceived  sensitivity  range  [6], 

Srinivasan  and  Basdogan  [8]  identified  multi-finger 
displays  as  an  important  future  development  in  the  field 
of  computer  haptics,  though  early  exponents  of  the  idea 
were  Atkinson  et  al.  [1],  who  outlined  an  idea  for  a 
"magic  glove"  which  used  miniature  gas  jets  to  stimulate 
the  users  hand.  It  was  also  proposed  by  Ledcrman  and 
Klatzky  [5]  that  multiple  end  effectors  may  be  important 
for  providing  additional  kinesthetic  information. 

We  have  adopted  a  design  strategy  incorporating  several 
"grounded"  interfaces,  as  opposed  to  an  "ungrounded" 


device,  such  as  a  glove,  or  exoskeleton,  as  such  devices 
have  previously  been  shown  to  impair  performance, 
whereas  it  is  widely  accepted  that  grounded  interfaces 
such  as  the  PHANToM  are  capable  of  supplying  realistic 
force  cues. 

For  example,  Turner  et  al.  [9]  present  results  pertaining  to 
a  system  for  dexterous  telemanipulation.  The  user  wears  a 
force  feedback  augmented  instrumented  glove 
(CyberGlove  ®  from  Virtual  Technologies)  which 
transfers  the  user's  movements  to  a  two  fingered 
dexterous  robot  hand  mounted  on  the  end  of  a  4-DOF 
industrial  robot  manipulator.  Forces  applied  to  the  fingers 
are  uni-polar,  serving  to  only  straighten  the  finger.  The 
results  indicated  that  the  single  axis  of  force  feedback 
presented  by  the  system  did  not  increase  speed  of 
performance  for  simple  telemanipulation  tasks.  Subjects 
pointed  out  that  the  force  feedback  could  be  misleading  in 
a  “block  stacking"  test,  whereas  provision  of  force  cues 
actually  had  a  negative  effect  on  performance  in  a  "knob 
turning"  test.  These  examples  serve  to  illustrate  the 
importance  of  the  device  used  to  display  force  cues,  and 
how  they  affect  task  performance  when  they  are  limited, 
unavailable  or  inappropriate  to  the  task  at  hand.  The  force 
feedback  system  had  a  cut  off  frequency  of  40  Hz,  which 
means  it  would  have  difficulty  displaying  higher 
frequency  slip  sensations  and  vibrations,  which  are 
important  in  manipulation  tasks. 
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Figure  1.  Workspace  of  three  PHANToM  composite  system. 


3.  Workspace  Configuration 

The  advantage  of  ungrounded  devices  is  that  they  are 
often  designed  with  regards  to  the  range  of  movement  / 
degrees  of  freedom  of  a  particular  human  joint  or  limb, 
such  as  is  the  case  with  hand  or  arm  exoskeletons. 
Ungrounded  devices,  such  as  the  PHANToM,  or  Impulse 
Engine  (www.immersion.com)  often  have  a  much  more 
limited  and  less  intuitive  workspace. 

As  the  composite  device  will  incorporate  several 
grounded  interfaces,  it  is  inevitable  that  their  combined 
workspace  will  amount  to  less  than  that  which  is  afforded 
by  a  single  device  of  that  type.  It  is  an  important  factor  in 
the  design  of  the  interface,  however,  to  consider  the  extent 
of  this  available  workspace,  and  how  this  will  facilitate 
employment  of  stereotypical  grasp  mechanisms  (e.g. 
pinch,  key  grasp  etc.),  and  EPs. 

Figure  1  illustrates  one  such  suitable  configuration  for  the 
three  PHANToM  devices.  Thus,  two  PHANToMs  are 
placed  opposite  each  other,  facing  each  other  in  the  reset 
position.  The  final  PHANToM  is  positioned  immediately 
below  the  first  two  devices,  and  is  inverted.  The 
workspace  is  roughly  20x30x15  cm.  A  typical 
configuration  for  a  two  PHANToM  3.0  system  offers  a 
workspace  of  30x40x40  cm  [10]. 


It  is  also  important  to  establish  the  range  of  flexion  and 
extension  available  to  the  fingers  and  wrist,  and  the  range 
of  grips  that  can  be  performed.  With  three  contact  points, 
it  is,  in  theory,  possible  to  perform  pinch  grasps,  tripod 
pinch  (pen  grasp)  and  key  grips.  However,  more 
investigation  is  required  to  fully  quantify  the  extent  to 
which  these  procedures  can  be  executed  under  a  given  3 
PHANToM  setup. 

4.  Future  Work 

It  is  planned  to  investigate  several  other  possible 
configurations,  in  order  to  ascertain  which  arrangement 
will  offer  the  best  workspace,  and  possibilities  for  user 
interaction  techniques.  A  similar  study  involving  two 
PHANToM  devices  highlighted  the  difficulties  involved 
in  calibrating  the  two  devices  [10].  Results  showed  that 
careful  calibration  of  the  two  devices  produced  an  average 
position  error  of  below  1mm  between  the  physical  point 
and  the  point  in  the  virtual  space.  Inevitably,  this  problem 
will  be  more  complex  with  3  devices,  and  potentially 
detrimental  to  the  illusion  of  physical  contact  and  grasp 
stability  in  the  simulated  environment. 

Typical  grasping  mechanisms  employed  by  users  have 
been  considered,  though  formal  verification  of  which 
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grasps  can  be  performed  and  their  efficiency  must  be 
investigated. 

5.  Conclusion 

This  paper  has  outlined  the  need  for  consideration  and 
development  of  multiple  contact  point  haptic  interfaces.  It 
was  illustrated  that  current  devices  tend  to  impair 
performance  of  stereotypical  EPs,  and  inhibit  acquisition 
of  information  regarding  object  properties.  This  mostly 
occurs  due  to  the  point  interaction  method  of  simulating 
contact,  though  the  devices  bandwidth  and  physical 
properties  are  also  a  contributory  factor.  It  is  envisaged 
that  a  device  incorporating  several,  grounded,  point 
interaction  type  devices  will  allow  more  natural 
exploration  of  simulated  environments,  potentially 
increasing  their  efficiency,  and  ease  of  use. 
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Abstract 

Despite  a  lot  of  research  in  recent  years  stable  and 
realistic  haptic  interaction  with  virtual  environments 
keeps  an  unsolved  problem.  Among  different  approaches 
the  Voxmap-PointShell™  (VPS)  method  seems  very 
promising  due  to  constant  sample  rates,  independent  of 
the  static  environment.  But  there  is  still  the  stability 
problem  to  be  solved  globally.  In  this  paper  some 
adaptations  based  on  the  VPS  are  presented,  including 
the  dynamic  object  modeling  and  the  force  calculation 
method,  to  reduce  the  distractions  of  the  calculated 
collision  forces  to  increase  stability.  The  PointShell  points 
lie  exactly  on  the  surface  of  the  dynamic  object,  to  get  a 
smoother  surface  representation.  A  variation  of  the 
collision  force  calculation  leads  to  a  reduction  of  the 
“ voxel  noise”,  that  appears  due  to  the  discretization  of 
the  volume  space.  A  design  framework  for  the  virtual 
coupling  is  presented,  that  enables  the  automatic 
configuration  for  different  haptic  devices.  The  validity 
will  be  shown  with  different  kineasthetic  hand  controllers. 


1.  Introduction 


environment)  with  the  dynamic  object,  and  the  rendered 
collision  forces  will  be  fed  back  to  the  operator. 

The  haptic  rendering  update  rate  is  independent  from 
the  complexity  of  the  static  environment,  thus  qualifying 
for  real  time  applications.  The  environment  of  static 
objects  is  collectively  represented  by  a  single  spatial 
occupancy  map  called  a  voxmap  (volume  map).  This  is 
created  by  discretizing  the  static  environment  space, 
which  is  partitioned  into  regions  of  free  space,  object 
surface,  and  object  interior.  The  collection  of  the  discrete 
volume  elements  (voxels)  build  the  voxmap.  The  dynamic 
object  is  described  by  a  collection  of  points  (PointShell), 
which  models  its  surface.  Each  point  is  assigned  a  surface 
normal  vector,  pointing  inwards.  The  haptic  rendering 
algorithm  includes  a  fast  collision  detection  technique 
based  on  probing  the  voxmap  with  the  surface  point 
samples  of  the  PointShell.  By  using  the  normal  vectors  of 
the  PointShell,  an  approximate  collision  force  can  be 
easily  computed  in  constant  time  for  each  point-voxel 
interpenetration.  Figure  1  shows  a  PointShell  colliding 
with  the  voxmap.  For  each  PointShell  point,  contained  by 
a  surface  voxel,  the  depth  of  interpenetration  is  calculated 
as  the  distance  d  from  the  point  to  the  tangent  plane.  This 
plane  passes  through  the  voxel  center  and  has  the  same 
normal  vector  as  the  PointShell  point  normal  vector. 


There  are  numerous  haptic  rendering  algorithms  for 
virtual  simulations,  differing  in  object  and  surface 
modeling.  Object  models  are  mostly  approximate 
descriptions  of  the  simulated  objects,  used  in  the  virtual 
environment.  In  many  rendering  algorithms,  the 
geometrical  complexity  of  the  particular  modeled  objects 
is  restricted  due  to  high  rendering  times.  In  the  VPS 
approach  of  McNeely,  Puterbaugh  and  Troy  [6],  the 
virtual  environment  consists  of  objects  divided  into 
dynamic  and  static  objects.  Dynamic  objects  can  freely 
move  through  the  virtual  space,  whereas  the  static  objects 
are  fixed  in  the  world  coordinate  system.  With  a  haptic 
device,  a  human  can  touch  the  static  objects  (static 
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So  the  time  consumed  to  render  a  single  frame  depends 
only  on  the  number  of  PointShell  points.  The  real  time 
capability  qualifies  this  algorithm  for  online  haptic 
interactions  between  a  human  operator  and  a  virtual 
environment.  For  enhancing  haptic  stability,  McNeely  et 
al.  [6]  employ  a  ‘virtual  coupling’  scheme,  which 
connects  the  user’s  haptic  motion  with  the  motions  of  the 
dynamic  object  through  a  virtual  spring  and  damper.  This 
scheme  was  embedded  in  the  haptic  rendering  algorithm. 
But  mechanical  properties  of  different  haptic  devices  are 
not  treated  by  this  virtual  coupling,  and  it  is  difficult  to 
tune  the  system  to  obtain  a  stable  haptic  feedback. 

In  contrast,  the  approach  described  in  this  paper 
decouples  the  stability  problem  from  the  haptic  rendering, 
as  also  proposed  by  Adams  and  I-Iannaford  [1],  where  the 
virtual  coupling  was  treated  as  part  of  the  haptic  interface 
(device  and  virtual  environment).  This  allows  the  usage  of 
an  arbitrary  haptic  rendering  algorithm  independent  from 
the  haptic  device.  Almost  no  changes  of  the  virtual 
environment  parameters  are  required,  when  the  haptic 
device  changes.  In  chapter  2,  we  present  adaptations  to 
the  VPS  based  object  modeling  and  force  calculation,  for 
reducing  the  distractions  of  collision  forces  to  get  more 
stability.  The  ‘virtual  coupling’  is  described  in  chapter  3. 
Chapter  4  shows  some  results  of  experiments,  where  the 
validity  of  the  methods  can  be  seen.  The  paper  closes  with 
the  conclusion  in  chapter  5. 

2.  Extending  the  VPS  Algorithm 

2.1  Exact  PointShell  Creation 


The  calculation  of  the  contact  force  takes  place  in  the 
force  voxel  layer  which  is  one  voxel  deep.  Due  to  the 
design  of  the  VPS  method  the  effective  force  layer  depth 
around  static  objects  amounts  to  half  a  voxel.  So  it  can  be 
seen  easily,  that  surface  deviations  of  the  PointShell 
object,  with  an  amplitude  of  more  then  a  half  voxel  size, 
have  considerable  influence  on  force  distractions.  This 
surface  variance  appears  due  to  the  PointShell  creation 
described  in  the  original  VPS,  where  the  object  first  was 
voxclized,  and  the  collection  of  all  surface  voxel  centers 
build  the  PointShell  (Fig.  3,  1-3). 

Next,  the  extended  PointShell  creation  is  presented, 
where  the  points  lie  on  the  triangulated  object  surface 
(anti-aliasing),  not  only  to  improve  the  accuracy,  but  also 
to  stabilize  the  collision  force.  The  example  in  Figure  2 
shows  the  same  collision  situation  of  a  dynamic  and  static 
object,  with  the  two  different  PointShell  models.  In  the 
upper  scene  (Fig.  2  a),  the  PointShell  was  built  with  the 
original  algorithm,  where  the  distance  between  the 


PointShell  points  and  the  object  surface  is  up  to 


where  e  denotes  the  voxel  extension.  This  surface 
deviation  is  very  high  compared  to  the  force  layer  depth 


of  e/2.  In  the  case  of  Figure  2  a)  the  PointShell  was 
created  using  the  voxel  raster  shown  with  dashed  lines. 
With  this  rough  surface  approximation  only  one  of  the 
four  PointShell  points  effects  a  collision  force,  although 
the  ‘real’  dynamic  object  surface  cross  three  force  layer 
voxel.  Figure  2  b)  shows  the  effected  collision  forces 
using  the  exact  PointShell  model,  creating  a  smoother 
force  field. 


Figure  2:  Different  collision  force  effects  in 
comparison  between  the  original 
PointShell  and  our  exact  Point  Shell 

The  exact  PointShell  creation  process  steps  are  shown 
in  Figure  3.  The  first  two  steps  are  equal  to  the  original 
PointShell  creation.  In  order  to  guarantee  a  proper 
detection  of  collisions  between  dynamic  and  static 
objects,  the  distance  between  two  neighbored  PointShell 
points  is  limited  according  to  the  voxel  resolution  of  the 
static  environment.  Hence  the  ‘naive’  PointShell  is  built 
by  the  sum  of  all  center  points  of  the  resulting  surface 
voxels,  after  creating  the  VoxMap  of  the  dynamic  object 
with  the  same  resolution  as  of  the  static  environment  (Fig. 
3,  1-3). 


I®* 

^^5 

%  t" 

Figure  3:  Steps  of  the  exact  PointShell 
creation  algorithm 


A  smoother  surface  representation  can  be  achieved  by 
projecting  each  center  point  on  to  the  triangle  surface 
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(Fig.  3,  4-5).  Two  additional  requirements  for  the  new 
position  have  to  be  fulfilled: 

1)  inside  the  primary  voxel 

2)  minimal  distance  to  the  voxel  center 

This  two  requirements  can  be  inconsistent  with  one 
another,  whereas  the  first  requirement  has  the  higher 
priority  and  the  second  should  be  optimized  as  far  as 
possible.  The  first  condition  keeps  the  maximal  neighbor 
point  distance  such  that  an  unrecognized  interpenetration 
of  the  static  and  dynamic  object  is  prevented.  The 
extension  of  the  spatial  gap  between  two  PointShell  point 
neighbors  is  restricted  to  the  minimal  spatial  extension  of 
a  static  object  surface  part  (3  voxel  extensions,  if  a  force 
field  is  used  as  described  by  McNeely  [6]).  The  property 
of  the  shortest  distance  tries  to  keep  the  original  distance 
of  the  point  neighborhood  as  far  as  the  other  conditions 
allows,  for  a  convenient  point  distribution. 

Algorithm:  The  following  algorithm  shows  the  correct 
point-surface-projection,  keeping  the  above  requirements 
with  the  defined  priority.  The  object  surface  is  described 
by  a  number  of  triangles.  Due  to  the  condition,  that  the 
projected  point  must  be  contained  by  the  respective  voxel, 
only  the  triangles  with  a  common  intersection  point  with 
that  voxel  have  to  be  considered.  For  each  triangle  the 
following  steps  have  to  be  done.  First  calculate  from  the 
voxel  center  the  perpendicular  nadir  p0  on  the  triangle 
surface.  Now  four  cases  have  to  be  considered: 

Let: 

-  V  be  the  set  of  all  points  contained  by  the  respective 
Voxel; 

-  VS  be  the  set  of  all  points  lying  on  the  surface  of  the 
respective  Voxel;  VS  c  V; 

-  T  be  the  set  of  all  points  lying  on  the  observed  triangle; 
then: 

case  1 :  p0  e  V  and  po  e  T: 
the  result  is  po; 

case  2:  po  e  V  and  p0  0  T: 

return  point  pr  e  { T  n  V } ,  which  has  the 
shortest  distance  to  p0; 

case  3:  po  0  V  and  p0  e  T: 

return  point  pr  e  {T  n  VS),  which  has  the 
shortest  distance  to  po; 

case  4:  po  0  V  and  p0  0  T: 

if  (pr  e  T  with  the  shortest  distance  to  p0)  e  V 
then  the  result  is  pr, 

else  the  result  is  equal  to  the  result  of  case  3; 

Figure  4  shows  two  examples  of  case  4.  The  treatment 
in  case  4  is  generally  valid,  therefore  all  cases  can  be 
handled  by  the  following  algorithm: 


Algorithm  : 

projectPoint(Voxel,  Triangle) 

p0  =  perpendicularNadir(Voxel.centerpt,  Triangle.plane); 

p,  =  trianglePointWithShortestDistanceToPoint(p„); 

if  (Voxel.contains(p,)) 

then  return  (p.) 

else  intersectionjines  =  intersect(Triangle,  Voxel. surface); 

for  each  Line  e  intersectionjines 
p,  =  linePointWithShortestDistanceToPoint(p0,Line); 
P™*  =  min(distance(p„p0),  dlstance(p^„p0)); 

return  preBU,.; 
end; 


For  each  original  PointShell  point  (voxel  center),  map 
it  on  the  corresponding  triangles,  and  choose  the  result 
point  with  the  shortest  distance  to  the  voxel  center.  The 
set  of  all  mapped  points  build  the  exact  PointShell.  Now 
the  PointShell  vectors  can  be  calculated  from  the  normal 
directions  of  the  triangles  intersecting  the  relevant  voxel. 


Triangle 

result  point 
(lying  on  voxel 
surface) 


result  point 
(lying  inside  the 
voxel) 


Figure  4:  Two  examples  showing  the 

projection  treatment  of  case  4 


2.2  Adapted  Collision  Force  Calculation 

Further  force  discontinuities  appear,  when  PointShell 
points  cross  static  voxel  borders.  Especially  under  sliding 
motion  with  a  relatively  constant  interpenetration  between 
the  dynamic  and  static  object,  this  effect,  shown  in  Figure 
5  (a,  b),  causes  instability  and  notably  disturbs  the 
operator’s  correct  sense  of  touch  (vibrations).  The 
presented  variation  of  the  VPS  collision  force  calculation 
[6],  leads  to  smoother  transitions  at  voxel  borders.  In 
contrary  to  hierarchical  methods,  which  increase  the 
calculation  time,  the  presented  method  does  not  need  a 
finer  resolution  (granularity)  of  the  static  environment. 

Instead  of  providing  a  general  solution  to  the  force 
stability  problem,  our  adapted  force  calculation 
specifically  targets  the  force  noise  for  dynamic  objects 
sliding  over  a  plane  of  adjacent  surface  voxels.  When 
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PointShell  points  cross  the  border  between  these  adjacent 
voxels,  instability  occurs  if  the  direction  of  the  relevant 
point  normal  vector  is  not  perpendicular  to  the  sliding 
plane.  Figure  5  a)  shows  a  dynamic  object  colliding  with 
a  static  object  under  sliding  motion.  Figure  5  b)  and  c) 
demonstrate  the  interesting  area  (marked  with  the  dashed 
line  in  5  a)).  Figure  5  b)  shows  the  problem  that  appears 
with  the  original  force  calculation  method.  When  the 
PointShell  point  crosses  the  voxel  border,  the  calculated 
force  value  is  reduced  to  zero,  which  influences  the 
overall  collision  force.  Instead  of  using  the  individual 
PointShell  normals  to  calculate  the  collision  force,  the 
average  of  all  collision  point  normals  is  used  to  calculate 
the  interpenetration  distance  d  of  each  PointShell  point 
(Figure  5  c)).  This  average  direction  is  a  more  stable 
approximation  of  the  sliding  plane  normal,  and  the 
mentioned  problem  can  be  mitigated. 


current  object 


without  with 

adapted  force  calculation 

Figure  5:  Force  stabilization  under  sliding  motion 
3.  Virtual  Coupling 

The  schemes  described  above  only  reduce  force  noise, 
but  there  are  still  remaining  distracting  forces  causing  an 
unstable  system.  The  direct  way  to  reach  the  required 
stability  is  to  use  a  ‘virtual  coupling’  scheme  between  the 
operator  and  the  virtual  environment.  A  design  framework 
for  the  virtual  coupling  is  presented,  that  allows  a 
consistent  interaction  between  the  haptic  rendering  (VPS) 
and  different  haptic  devices,  because  for  the  overall 
stability,  the  dynamic  properties  of  the  haptic  device  play 
a  key  role. 

In  many  other  approaches  (e.g.  McNeely  et  al.  [6], 
Adams  and  Hannaford  [1])  the  virtual  coupling  is  treated 
and  parameterized  as  a  spring-damper  system,  which  is  a 
good  mechanical  interpretation,  but  leads  in  most  cases  to 
a  heuristic  optimization  of  the  parameters.  The  spring- 


damper  system  is  used  to  reduce  the  force  steps,  produced 
by  discontinuities  and  discretization,  for  the  human  user. 

The  virtual  coupling  can  also  be  seen  as  a  dynamic 
shaping  filter,  as  it  is  done  in  this  approach  (Fig.  6).  The 
dynamics  of  the  calculated  contact  force  are  reduced  such 
that  neither  the  haptic  device  nor  the  force  reflection  loop 
with  the  human  become  unstable. 


Figure  6:  Signal  flow  diagram  of  the  haptic 
simulation 

Filter  Construction 


The  filter  should  keep  the  role  of  the  virtual  coupling, 
thus  the  physically  model  of  the  virtual  coupling  (the 
spring-damper  system)  is  used  as  a  model  for  the 
construction  part. 

F 

— -► 

c  m 

Figure  7:  Mechanical  spring-damper  system 

With  this  mechanical  representation  (Fig.  7)  the 
following  equation  can  be  built: 

F  =  C'X  +  r-x  +  m-  x 


For  the  filter,  the  translation  value  x  must  be  replaced 
by  the  collision  force  as  input,  which  is  generated  by  the 
haptic  rendering  algorithm.  This  replacement  can  be  done 
under  the  condition  that  the  collision  force  depends 
directly  on  the  penetration  distance,  which  is  true  with 
this  haptic  rendering  algorithm  (Fco!t  ~  Xpos  (collision))* 

With  T  =  J—  ,  w0  =  ~ ,  d  -  — p= r ,  K  =  - 
V  c  T  2  c 

this  function  leads  to  the  PT2-element  with  the  following 
transfer  function  (continuous  time): 


Hf(s)  =  K ■ 


wl  +  2  •  d  • 


The  continuous  time  filter  can  be  transformed  to  the 
discrete  time  filter  IIf(z)  with  the  Tustin-Approximation: 


HAz)  =  K- 


vt;Q  +2  ■  d  • 


_2_  1  -z~‘  |  4  (1  -z'1)2 
TA'l  +  z'1  7’a2  (1  +  z-1)2 
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The  parameters  used  with  this  filter  function  can  easily 
adjusted  to  the  dynamic  properties  of  the  haptic  device, 
without  changes  in  the  haptic  rendering  algorithm. 
Whereas  the  sample  time  TA  is  chosen  by  the  cycle  time 
of  the  rendering  algorithm  and  haptic  device,  the 
sufficient  stability  can  be  reached  by  adjusting  the  cut-off- 
frequency  Wo  and  the  damping  factor  d. 

Typically  a  human  can  sense  kineasthetic  impression 
with  a  bandwidth  of  approximately  lkllz  [3]  and  can 
make  controlled  movements  with  a  maximal  bandwidth  of 
10  Hz.  Studies  at  the  DLR  resulted  in  a  bandwidth  of  2 
Hz  in  which  a  human  performs  a  desired  task  (no 
reflexes).  To  perceive  realistic  kineasthetic  impression  the 
dynamic  of  the  force  controller  (implemented  at  the 
device)  plays  the  key  role  and  limits  the  achievable 
bandwidth.  That  means  that  the  force  dynamic  of  the 
virtual  environment  can  be  limited  to  5Hz  without  loosing 
a  realistic  impression  of  the  scene. 

4.  Experiments 

The  validity  will  be  shown  in  experimental  results 
using  the  Phantom  T-Model  (Fig. 14)  and  the  DLR-  light 
weight  robot  (Fig.  13)  as  kineasthetic  hand  controller  [7]. 
In  the  virtual  test  bed  the  static  scene  consists  of  a  cup, 
whereas  a  sphere  is  used  as  dynamic  object  manipulated 
by  the  operator  (Fig  15). 

4.1  Experiments  to  the  Adapted  Collision  Force 
Calculation 

For  a  benchmark  of  the  force  distractions,  the  results 
of  the  collision  force  calculations  are  evaluated  in  two 
experiments.  The  dynamic  sphere  was  moved  over  the 
static  object  with  a  predefined  path  and  a  constant 
penetration.  The  experiments  differ  with  respect  to  the 
surface  conditions  of  the  static  object,  which  typically 
occur  in  discrete  space  models  like  the  voxel  map.  The 
surface  condition  depends  on  the  orientation  of  the 
surface  plane  related  to  the  main  axis  planes  of  the  voxel 
grid  coordinate  system.  Force  distractions,  appearing  in 
sliding  motion,  primarily  influence  the  collision  force 
direction.  For  demonstrating  this  effect,  only  the  force 
parts  vertical  to  the  collision  plane  normal  (the  intuitive 
collision  force  direction)  were  recorded.  In  each 
experiment  this  record  is  done  for  both  the  original  and 
the  adapted  collision  force  calculations.  The  direct 
comparison  between  them  is  depicted  in  the  following 
diagrams,  made  for  each  experiment. 

Experiment  1:  sliding  motion  over  a  plane  surface 

The  sliding  path  of  the  dynamic  object  is  in  this 
experiment  a  circle  movement  over  a  plane  surface  of  the 
static  object.  In  Figure  8  all  the  distraction  arrowheads  are 


recorded,  by  keeping  the  axis  origin  at  the  sphere  center 
position. 

Force  dlstract'on  Plol  {without  adaptation)  Forco  distraction  Plot  (with  adaptation) 


Figure  8:  Collision  force  distractions  of 
experiment  1 

In  the  experiment  related  to  Figure  8  a)  and  b)  the 
sliding  surface  of  the  static  object  was  oriented  along  the 
voxel  grid  main  axes,  whereas  8  c)  and  8  d)  presents  the 
results  of  the  motion  over  a  not  voxel  grid  raster  oriented 
plane.  With  the  adapted  force  calculation  the  distraction 
reduction  is  appreciable  in  both  cases. 

Experiment  2:  sliding  motion  over  a  curved  surface 

In  this  experiment  the  sphere  slides  on  a  circle  path 
around  the  superficies  surface  of  a  cylinder,  with  also  a 
constant  penetration  depth  into  the  cylinder  (Fig.  10).  This 
shows  the  collision  force  distraction  behavior  with  many 
different  surface  condition  cases.  Figure  9  a)  shows  the 
distractions  of  the  collision  force  over  the  time  calculated 
with  the  original  algorithm  and  Figure  9  b)  shows  the 
distractions  with  the  adapted  algorithm.  Of  cause  there  are 
still  remaining  distractions  with  the  adapted  form  coming 
from  the  different  surface  conditions,  but  a  high  reduction 
of  these  distractions  can  be  seen  with  the  adapted 
collision  force  calculation. 


Figure  9:  Collision  force  distractions  in  one 
direction  of  experiment  2 


Figure  10:  Benchmark  for  the  virtual  coupling 
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4.2  Experiments  to  the  Virtual  Coupling 

To  evaluate  the  virtual  coupling,  a  sphere  is  moved 
over  a  cylindrical  static  object  with  a  constant  penetration 
(Fig.  10). 

fotc«  with  vlrlvn' Kfwpltng  F  v<-  ■.*  ">  ■ ,  <  i  ij ' 


Figure  1 1 :  Force  plots  for  different  virtual 
couplings 

In  Figure  11  shows  the  results  of  a  heuristic  tuned 
virtual  coupling  and  the  dynamic  shaping  filler  explained 
in  chapter  3.  In  hard wareTn-the- loop  experiments,  the 
virtual  cup  (Fig.  15)  was  touched  by  the  virtual  sphere 
using  the  DLR  Light-Weight  robot  (I .Wk)  (big.  13)  and 
the  Phantom  T-Modcl  (Fig.  14)  as  haptic  hand  controllers. 
The  following  diagrams  (Fig.  12)  shows  the  collision 
forces  (thin  line)  calculated  by  the  haptic  rendering 
algorithm  and  the  corresponding  output  force  (thick  line), 
which  was  returned  as  feedback  force  to  the  hand 
controllers  by  the  virtual  coupling  element.  Two  kinds  of 
collisions  were  tested,  an  abrupt  collision,  hitting  the  cup 
with  the  sphere  (Fig.  12  a,e),  and  a  steady  collision, 
sliding  with  the  sphere  over  the  cup  surface  (Fig.  12  b,d). 


Figure  12:  Force  plots  of  the  experiments  with 
the  Phantom  and  the  L.WR 


5.  Conclusions 

In  this  paper  some  efforts  wore,  done  to  improve  the 
stability  and  immersion  of  haptic  feedback.  Our  approach 
is  based  on  the  VPS  method,  due  to  its  constant  sample 
rates.  First  the  object  modeling  was  impmved  towards  an 
exact  surface  representation.  This  not  only  increases  the 
accuracy,  but  also  avoids  some  torce  distractions  during 
contact  situation  with  static  objects.  The  force  calculation 
itself  was  adapted  such  that  the  voxel  noise,  occurring 
during  sliding  motions  over  static  objects,  is  substantially 
reduced.  This  leads  to  a  smoother  force  evolution  within 


the  task.  Finally,  a  design  framework  for  virtual  coupling 
was  presented.  It  takes  into  account  the  dynamic 
properties  of  the  human  operator  and  of  the  hand 
controller.  This  approach  leads  to  a  more  stable  haptic 
interaction  with  virtual  environments  as  it  is  shown  in  the 


presented  experiments. 

Further  work  in  the  fields  of  improving  the  realistic 
haptic  feedback  extending  the  VPS,  and  of  automatic 


design  of  the  virtual  coupling  will  be  done. 


Figure  13: 

DLR  Light-Weight  robot 
as  kineasthetic  hand 
controller 


Figure  14: 
Phantom  T-Model 
from  Sensable 
Technologies  Inc. 


Figure  15: 
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Abstract 

We  describe  a  calibration  technique  for  a  hand-immersive 
virtual-reality  environment  -  the  Reach  in  Display  [I]. 
The  technique  we  present  allows  us  to  calibrate  both  the 
haptics  device  and  the  stereo  display  environment 
simultaneously  where  calibration  of  the  stereo  display 
utilises  the  haptics  device  and  vice  versa.  The  calibration 
procedure  is  described  and  we  present  a  modular 
architecture  for  constructing  calibration  functions  and 
show  how  fine  control  over  the  individual  calibration 
parameters  that  are  used  can  he  obtained. 


1.  Introduction 

The  Reachin  Display  [I]  comprises  yu  articulated  haptics 
device  [2]  and  a  stereoscopic  visual  display.  The  working 
volumes  of  these  two  components  are  arranged  to  coincide 
so  as  to  effect  a  co-located  hapto-vistial  display,  i.e.  the 
user  can  feel  the  object  in  the  same  place  as  he  sees  it. 
This  is  accomplished  by  using  a  semi  transparent  mirror  to 
reflect  the  scene  being  drawn  giving  the  effect  that  the 
objects  appear  in  space  in  the  same  place  as  the  hands  and 
the  haptics  device  (see  Figure  L).  The  greatest  benefit  of 
such  a  system  is  that  it  allows  the  user  to  utilise  their  sense 
of  proprioception  to  quickly  and  accurately  locate  objects 
within  the  workspace.  Also  there  is  a  much  stronger  sense 
of  realism  than  the  traditional  set-up. 

Co-located  displays  require  accurate  calibration  in 
order  for  the  user  to  perceive  the  combined  haptic  and 
visual  displays  as  a  single  multi-sensory  environment. 
Also,  if  the  mirror  in  the  display  is  semi-transparent  then 
the  user  secs  both  the  real  objects  in  the  workspace  (such 
as  their  hands)  as  well  as  the  virtual  objects  displayed  by 
the  monitor.  In  this  augmented  reality  mode  (which  is 
used  during  calibration)  any  error  in  calibration  will  be 
obvious  and  disturbing.  We  would  like  to  be  able  to 
acquire  an  accurate  calibration  in  an  easy  way  with  as  little 


work  needed  from  the  user  as  possible.  For  example,  the 
task  of  setting  the  parameters  manually  requires  a  lot  of 
work  and  any  change  to  the  set-up  invalidates  the 
calibration.  Since  the  set-up  can  vary  quite  a  lot  we  would 
like  to  be  able  to  generate  the  calibration  automatically. 


Figure  1:  User  using  the  Reachin  Display.  The  user  looks 
through  the  mirror  and  thereby  experiences  the  objects  in  the 
same  space  as  his  hands. 

The  calibration  actually  consists  of  two  parts.  One  part 
is  the  translation  of  the  arms’  position  of  the  haptics 
device  into  a  position  in  the  stereo  display.  This  is  not 
trivial  since  the  haptics  device  in  theory  can  be  mounted  in 
any  position  and  orientation  on  the  display.  Hence  a 
movement  to  the  left  in  the  haptics  device  coordinate 
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space  can  be  a  movement  in  any  direction  in  the  stereo 
display  environment.  The  other  thing  that  needs  to  be 
calibrated  is  the  visual  display  of  the  scene.  We  need  the 
objects  to  be  the  right  size  (the  size  specified  by  the  user) 
and  the  perspective  to  be  right  so  that  the  stereo 
environment  does  not  differ  (torn  the  real  world.  I  his 
means  that  a  real  object  and  a  virtual  object  of  the  same 
size  should  appear  to  have  same  size  anywhere  in  the 
stereo  display  work  area. 

We  present  a  calibration  technique  where,  both  these 
parts  can  be  calibrated.  The  calibration  of  the  stereo 
display  environment  utilises  the  haptics  device  and  the 
calibration  of  the  haptics  device  utilises  the.  stereo  display 
environment. 

2.  Calibration  procedure 

During  the  calibration  process  the  user  sees  both  the  real 
stylus  of  the  haptics  device  as  well  as  a  virtual  model  of 
the  stylus  moving  within  the  stereoscopic  display.  When 
everything  is  calibrated  correctly  these  two  objects  should 
coincide  accurately  in  position,  orientation  and  size.  When 
they  do  not,  the  user  may  click  the  button  on  the  stylus  to 
freeze  the  virtual  stylus,  then  move,  the  real  stylus  so  that  it 
coincides  accurately  with  the  virtual  stylus  and  then 
release  the  button.  This  records  a  collection  .sample.  1’his 
sample  consists,  in  our  case,  ol  the  encoder  angles  ol  the 
arms  of  the  haptics  device  and  the  screen  coordinates  for 
the  left  and  right  eye  of  the  position  of  the  virtual  stylus. 
The  user  can  specify  several  correction  samples  and  these 
samples  are  then  used  by  the  software  to  calibrate  some 
subset  of  more  than  12  independent  calibration 
parameters. 

3.  Calibration  functions 


zero  and  therefore  the  relative  encoder  angles  also  are  the 
absolute  encoder  angles.  However  this  reset  position  is  not 
always  reachable  by  the  haptics  device  (e.g.  the  mirror  in 
the  display  can  be  in  the  way)  and  even  if  it  is  it  can  be 
quite  hard  to  put  it  in  the  reset  position  with  good 
accuracy.  So  instead  of  doing  this  we  include  the  reset 
angles  in  the  calibration,  which  eliminates  all  such 
problems. 

These  encoder  angles  may  then  be  transformed  into  a 
Cartesian  position  and  stylus  orientation  according  to 
known  geometry  information  such  as  the  lengths  of  the 
haptics  device  arms.  This  position  is  then  further 
transformed  into  the  world  coordinates  of  the  rendering 
software  by  means  of  a  calibration  translation,  rotation  and 
scaling.  This  is  the  coordinate  space  that  a  user  of  the 
Rcachin  Display  uses  to  specify  objects  in. 

Finally  the  world  coordinates  are  projected  onto  the 
stereoscopic  screen  with  one  position  for  the  left  eye  view 
and  one  for  the  right  eye  view.  This  transformation  is 
controlled  with  OpenGL  [4j[7]  parameters  and  can  be 
divided  into  several  parts.  First  the  world  coordinates  are 
transformed  by  a  model  view  matrix  into  eye  coordinates. 
The  eye  coordinates  are  the  coordinates  of  the  objects 
looking  into  the  world  through  a  camera  with  a  specified 
position  and  orientation.  Since  we  use  a  stereoscopic 
display  we  must  make  two  such  transformations,  one  with 
the  camera  in  the  position  of  the  left  eye  and  one  with  the 
camera  in  the  position  of  the  right  eye.  After  this 
transformation  a  projection  matrix  is  applied  and  the 
coordinates  are  normalised.  The  projection  matrix  is  set  up 
by  specifying  an  asymmetric  view  frustum  for  each  eye. 
The  view  frustum  is  constructed  from  field  of-view  angles, 
focal  distance  and  inter-ocular  distance  (see  Figure  2). 
This  gives  us  the  screen  coordinates  for  the  left  and  right 
eye  (also  called  normalised  device  coordinates)  and  this  is 
the  end  product  of  our  calibration  function  since  these 
define  the  perceived  3-dimensional  position  in  the  display. 


A  calibration  function  is  a  multi-valued  vector  1  unction 
from  a  set  of  values  (called  real  values)  and  a  set  of 
calibration  parameters  to  another  set  of  values  (called 
virtual  values).  When  the  user  clicks  the  button,  they 
specify  the  virtual  values  ol  the  correction  sample  and 
when  they  release  the  button  they  specify  the 
corresponding  real  values. 

In  our  case  the  calibration  function  consists  of  several 
stages.  We  begin  with  relative  encoder  angles  read 
directly  from  the  encoders  on  the  arms  ol  the  haptics 
device.  In  our  application  there  arc  3  encoders  measuring 
position  and  3  for  orientation  of  the  stylus.  These  angles 
arc  relative  to  the  angular  positions  of  the  encoders  when 
the  haptics  device  was  last  sent  a  reset  command. 
Therefore  we  offset  these  values  by  the  6  reset  angle 
calibration  parameters  to  calculate  the  absolute  encoder 
angles.  Normally  the  device  has  a  standard  reset  position 
in  which  the  reset  command  is  supposed  to  be  called  at. 
This  defines  the  position  where,  the  reset  angles  are  all 


Focal  distance 


Figure  2:  The  asymmetric  view  frustum  for  the  left  eye.  The 
centre  view  is  the  normal  view  when  stereo  is  not  used  and 
lire  view  from  the  left  eye  is  the  corresponding  view  used  in 
stereo  mode  for  the  left  eye.  A  similar  frustum  is  used  for  the 
right  eye. 
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We  then  define  an  error  function  based  on  the 
discrepancy  between  the  measured  real  values  transformed 
by  the  calibration  function  and  the  measured  virtual 
values.  Specifically  the  discrepancy  is  defined  to  be  the 
distance-squared  between  the  virtual  values  and  the 
transformed  real  values.  So  the  error  for  a  correction 
sample  is 

e(r,v,c)  -  (f(r,c)  -  r)' 

where  r  are  the  real  values  of  the  correction  sample,  r  are 
the  virtual  values,  c  are  the  calibration  parameters  and /is 
the  calibration  function,  if  we  have  several  correction 
samples  the  total  error  is  the  sum  of  the  errors  of  the 
individual  samples. 

This  gives  us  a  single  error  value  and  the  task  of  the 
calibration  procedure  is  then  to  adjust  nominated 
calibration  parameters  so  as  to  minimise  this  error 
function.  This  is  essentially  a  numerical  minimisation 
function  in  multiple  dimensions.  Wc  use  a  variable  metric 
method  to  perform  the  minimisation.  The  method  used  is 
the  Broyden-Fletcher-Goldfaib  Shanno  variant  of  the 
Davidon-Fletcher-Powell  minimisation  algorithm  [3].  This 
method  requires  us  to  provide  the  gradient  of  the  function 
to  minimise. 

Let  J,(r,c)  be  the  matrix  whose  columns  are  the  partial 
derivates  of/ with  respect  to  the  calibration  parameters  c. 
This  is  a  sub-matrix  of  the  full  Jacobian  of  f  Using  this 
function  wc  may  define  the  error  gradient  with  respect  to 
the  calibration  parameters  c  as 

G(r,v,c)  =  2  [/(/-,<•)- 1’]  /  (r,c) 

This  is  a  sub-vector  of  the  full  gradient  of  e,  containing 
only  the  partial  derivatives  of  a  with  respect  to  c.  In  the 
case  that  the  error  is  summed  over  several  correction 
samples,  the  error  gradient  is  simply  the  sum  of  the 
gradients  for  each  sample.  How  to  get  hold  of  the  Jacobian 
function  is  explained  in  the  next  section. 

The  problem  of  the  minimisation  is  that  it  is  easy  to 
find  a  local  minimum  in  the  error  function  and  hence  wc 
might  not  get  the  optimal  solution.  In  order  to  prevent  this 
we  make  several  minimisations  from  different  starting 
points  and  choose  the  smallest  minimum  wc  can  find  of  all 
the  minimisations.  Of  course  this  does  not  always  find  the 
global  minimum  either,  but  at  least  we  have  improved  our 
chances  to  get  a  better  minimum  than  if  wc  had  just  used 
the  first  one  we  found. 

5.  Calibration  modules 

When  implementing  the  calibration  functions  into  a 
calibration  program  we  would  like  to  be  able  to  make 


changes  to  them  easily.  We  would,  for  example,  like  to  be 
able  to  choose  which  calibration  parameters  to  calibrate 
and  which  to  leave  unchanged.  We  would  also  like  to  be 
able  to  extend  a  calibration  function  with  extra  parameters 
without  having  to  rewrite  any  code  from  the  previous 
function.  To  accomplish  this  we  use  a  number  of 
calibration  modules  to  construct  the  calibration  function. 
A  calibration  function  can  consist  of  one  or  more  of  these 
calibration  modules.  If  we  want  to  do  a  change  to  the 
calibration  function  (c.g.  add  a  rotation)  we  just  construct 
a  new  calibration  module  defining  this  transformation  and 
insert  it  into  the  calibration  function. 

A  calibration  module  defines  a  function  /from  a  set  of 
values  A  to  another  set  of  values  B.  The  values  of  A  can  be 
divided  into  input  parameters  and  calibration  parameters. 
The  calibration  parameters  are  the  parameters  that  are  to 
be  adjusted  to  get  the  best  calibration  while  input 
parameters  are  the  positional/geometric  values  to  be 
transformed  by  the  module.  For  example,  if  the  module 
specified  a  translation  in  Cartesian  coordinates  it  would 
take  six  parameters.  Three  of  them  would  be  calibration 
parameters  specifying  the  translation  at  each  axis  and  three 
would  be  input  parameters  specifying  the  original 
coordinates  which  we  want  translated. 

Each  module  also  defines  two  Jacobian  functions.  One 
of  the  functions,  J„  returns  the  Jacobian  matrix  for  /with 
respect  to  the  input  parameters  and  the  other  function,  /, 
returns  the  Jacobian  matrix  of  /  with  respect  to  the 
calibration  parameters.  These  Jacobian  functions  can  be 
quite  complicated  and  in  many  cases  they  would  be 
intractable  to  code  by  hand.  Because  of  this  we  use  the 
symbolic  mathematics  software  Maple™  to  automatically 
generate  the  code  for  the  calibration  modules.  This 
software  allows  us  to  specify  the  function  of  the 
calibration  module  symbolically.  Partial  derivatives  and 
Jacobian  matrices  can  then  easily  be  generated  with  the 
software  and  we  use  the  automatic  code  generation  tool  to 
get  optimised  functions  for  evaluating  both  the  Jacobian 
function  and  the  normal  function  of  the  calibration 
module.  This  way  the  calibration  modules  become  “black 
boxes”  of  code,  which  can  be  composed  into  larger 
calibration  modules,  which  allows  for  an  efficient  and 
extensible  run-  time  implementation. 

When  two  of  these  modules  have  been  defined  we  can 
compose  them  into  one  single  calibration  module 
performing  the  composed  transformation  of  the  two 
individual  modules.  The  only  restriction  is  that  the  number 
of  output  parameters  of  the  first  module  matches  the 
number  of  input  parameters  of  the  other  module.  When 
composing  two  calibration  modules  into  a  single  one  we 
must  create  composed  versions  of  the  function  and  the  two 
Jacobian  matrices.  For  the  function  value  it  is  quite 
straightforward. 

/V.c")  =  f'\r,c  ]  c')  =  f'{f(r,c),c ) 
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where  /is  the  function  of  the  first  module  and  r  are  the 
calibration  parameters  for  the  first  module.  I  lie  primed 
versions  arc  same  but  for  the  second  module,  r  is  the  input 
parameters  for/!  Note  that  the  calibration  paiamclcts  lot 
this  new  function  are  the  composition  of  the  two  vectors  of 
the  calibration  parameters  for  the  individual  modules. 

To  derive  the  composite  .lacohian  matrices  we  use  the 
chain  rule  for  vector  valued  lunctions.  1  lie  Jacobian 
function  with  respect  to  the  input  parameters  is  given  by 

~  \  c)  ~  J’{fU\c),c')xJl{r,c) 

where  ./,,  f,  and  c  arc  the  Jacobian  function,  the  function 
and  the  calibration  parameters  lor  the  lirst  module 
respectively.  The  primed  versions  are  the  same  but  lor  the 
second  module.  Note  that  the  matrix  multiplication 
enforces  the  restriction  that  the  number  of  input 
parameters  of  the  second  module  must,  be  the  same  as  the 
number  of  output  parameters  of  the  lirst  one.  The  Jacobian 
function  with  respect  to  the  calibration  parameters  is 
acquired  by 

J'(r,c')  =  .r. (V,r|r')  --  |/\  |  ttj 

where 

A  -  ./'(./ Xr,c),c')x  ■) ,.(r>t:) 

b  =  ./;(/(/■, <),c  0 

In  this  way  we  can  compose  as  many  modules  as  we 
want  into  the  final  calibration  function  and  it  is  easy  to 
make  changes  to  it  since  we  can  just  icmovc  or  add  the 
calibration  modules  that  we  would  like  to  use.  If  we  want 
the  calibration  parameters  for  a  certain  module  to  remain 
the  same  during  a  calibration  we  just  have  to  return  a  zeto 
matrix  as  This  way  vve  can  easily  control  which 
parameters  to  calibrate. 

6.  Implementation  and  results 

We  implemented  the  calibration  program  in  the  Reachin 
API  |6]  for  use  in  the  Reachin  Display.  We  used  the 
Python  script  language  L-HIbl  f°r  die  implementation 
according  to  the  mode!  described,  lo  make  it  tun  lastei 
the  black  boxes  of  the  calibration  modules  were 
implemented  in  C++. 

Wc  found  that  it  was  not  good  trying  to  calibrate  all 
parameters  at  once  unless  the  calibration  from  the  start  is 
acceptable.  In  our  tests  the  minimisci  only  I i n cl s  local 
minima  with  quite  large  errors  if  vve  do  this.  I  his  icsulls  in 

some  very  bad  calibrations.  We  lound  that  the  best 

calibration  was  acquired  by  first  calibrating  only  the 

haptics  device  (transformation  between  encoder  angles 
and  world  coordinates)  and  not  change  the  OpcnGL 

parameters.  Most  of  the  time  this  is  enough  since  the 


default  standard  OpcnGL  parameters  provided  with  the 
Reachin  Display  are  general  enough.  However  to  achieve 
a  better  calibration  and  get  the  sizes  to  match  better  we 
can  use  the  new  calibration  as  starting  point  and  let  the 
minimiser  find  a  minimum  from  there.  This  usually  results 
in  the  best  calibrations. 

If  this  strategy  is  used  the  result  is  good.  We  get  a 
precision  to  within  a  few  millimetres  in  the  recommended 
work  area  of  the  haptics  device.  The  precision  outside  this 
area  is  also  good  but  differs  a  little  depending  on  the 
sample  set  used.  As  a  worst  case  the  calibration  can  be  off 
by  a  centimetre  but  this  is  usually  in  the  extreme  positions 
of  the  haptics  device  and  does  not  really  matter  since  it 
should  not  be  used  in  these  positions  anyway.  To  get  a 
good  calibration  at  least  5  correction  samples  are  needed. 
Any  less  than  that  the  calibration  is  fine  at  the  sample 
points  but  is  poor  in  the  space  between  them.  It  is  also 
important  to  have  the  samples  spread  out  to  cover  as  much 
as  possible  of  the  work  area,  otherwise  the  calibration 
could  get  bad  in  the  area  without  any  sample. 

7.  Discussion  and  conclusions 

The  simple  calibration  utility  makes  it  easy  to  make 
individual  calibration  files  for  each  user  of  the  device. 
This  is  preferred  to  having  a  general  calibration  for  all 
users.  The  problem  of  not  having  individual  calibrations  is 
that  if,  for  example,  two  people  of  different  heights  were 
to  use  the  display  they  would  have  a  different  distance 
from  the  eyes  to  the  mirror  and  hence  the  same  calibration 
would  appear  dif  ferent  for  each  of  them.  To  make  it  work 
they  both  have  to  have  their  head  in  the  same  position  and 
have  the  same  inter-ocular  distance,  which  will  mean  that 
at  least  one  of  the  users  will  experience  poor  calibration. 

'I  he  calibration  can  never  be  better  than  the  sample  set 
given  to  the  calibration  routine.  Since  these  samples  are 
based  on  a  user’s  visual  experience  there  is  bound  to  be 
errors  while  specifying  the  samples.  It  is  impossible  to  get 
the  samples  exactly  right  and  hence  the  calibration  will 
always  have,  this  extra  error  measure  which  it  cannot 
compensate  for.  The  calibration  will  always  be  adversely 
affected  by  any  poor  samples  given  by  the  user. 

Currently  the  biggest  problem  is  finding  a  global 
minimum  in  the  presence  of  many  local  minima.  The 
current  strategy  of  randomly  restarting  minimisation 
sometimes  results  in  slow  performance  that  docs  not 
always  find  the  global  minimum.  In  future  we  intend  to 
explore  alternative  minimisation  routines  that  are  better 
able  to  cope  with  these  types  of  functions.  Nevertheless 
the  performance  of  the  current  system  is  satisfactory  for 
our  purposes  since  calibration  only  needs  to  be  performed 
occasionally  but  should  be  as  accurate  as  possible. 
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Abstract 


W e  present  a  method  to  compute  the  interaction  force 
during  an  imposed  movement.  First,  i vc  show  how  to 
incorporate  a  kinematic  constraint  into  the  equation  of 
motion  of  a  simple  but  realistic  model  of  the  arm.  Second, 
we  apply  this  method  to  predict  the  interaction  force  during 
an  imposed  movement  of  the  arm,  The  results  of  these 
simulations  are  compared  to  the  interaction  force  recorded 
when  a  robot  moved  the  hand  of  blindfolded  subjects 
according  to  a  predetermined  trajectory.  The  most  salient 
characteristics  of  the  interactionforce  could  be  reproduced 
by  assuming  that  the  subjects  supported  the  weight  of  their 
arm  without  attempting  to  anticipate  or  resist  the  imposed 
movement. 


1.  Introduction 

While  the  subject  of  compliant  motion  has  generated  a 
huge  literature  in  robotics  [  1 ,2],  remarkably  little  is  known 
about  how  humans  control  their  movements  when  they  must 
comply  with  an  external  constraint.  In  addition,  most  of  the 
studies  with  human  subjects  dealt  only  with  passive  external 
constraints  such  as  a  rigid  surface  |3).  For  example, 
McIntyre  et  al.  [4]  examined  whether  subjects  monitored 
the  force  through  sensory  feedback  or  relied  on  the 
compliant  properties  of  the  arm  to  slide  a  hand  along  a  rigid 
surface.  In  contrast,  our  study  focuses  on  the  interaction 
force  arising  during  an  imposed  movement  of  the  arm.  Such 
movements  occur  when  two  humans  shake  hands  together 
or  when  robots  are  used  in  the  rehabilitation  of  stroke 
patients  (51. 

The  interaction  force  in  such  a  situation  depends  on 
three  factors  at  least:  the  kinematic  characteristics  of  the 
imposed  motion,  the  mechanical  properties  of  the  arm,  and 
the  activity  of  the  subject  (e.g.  reflex  or  voluntary 
contraction  of  the  muscles  anticipating  or  resisting  the 
imposed  motion).  The  main  objective  of  this  paper  is  to 
describe  a  method  to  assess  the  relative  importance  of  these 
factors  in  the  interaction  force. 

In  the  Methods,  we  show  how  to  incorporate  a  kinemalic 
constraint  on  the  hand  position  into  the  equation  of  motion 
of  a  simple  but  realistic  model  of  the  arm  to  predict  the 
interaction  force  during  an  imposed  movement.  We  also 
describe  a  procedure  to  correct  errors  during  the  integration 


A  B 


Figure  1.  A:  A  schematic  view  of  the  hand  movement  imposed  by 
the  robot,  li.  Depiction  of  the  four  degrees  of  freedom  of  the  arm 
model  used  in  the  simulations  (Op.  elbow  azimuth,  02:  elbow 
elevation,  0,:  humeral  rotation,  Op  elbow  flexion). 


which  may  cause  the  simulations  to  diverge  from  the 
constraint  over  a  long  period  of  time.  In  the  Results,  we 
apply  this  method  to  predict  the  interaction  force  under  two 
different  hypotheses  about  the  activity  of  the  subjects  during 
an  imposed  motion.  The  results  of  these  simulations  are 
compared  to  the  interaction  force  recorded  while  a  robot 
moved  the  hand  of  blindfolded  subjects  (Fig.  1  A).  Finally, 
we  briefly  discuss  what  these  preliminary  results  suggest 
about  the  activity  of  the  subjects  in  this  task. 

2.  Methods 

The  Arm  Model.  The  arm  was  modeled  as  two  connected 
cylindric  bodies  (Fig.  IB).  The  first  body  (upper  arm)  was 
connected  to  the  torso  by  a  spherical  joint  (shoulder)  with 
three  degrees  of  freedom  (elbow  azimuth,  elbow  elevation 
and  humeral  rotation).  The  second  body  (forearm)  was 
connected  to  the  first  one  by  a  hinge  (elbow)  joint  with  one 
degree  of  freedom  (elbow  flexion). 

The  equation  of  the  motion  of  the  arm  model  is: 

/TO.  6, 0  +  H.0  A<)  +  C(0)  = 

Af(0)d  +  /bo)  66  -  C(0)()2 

where  o  —  f o ,  «>  o,  o4 j 1 ,  o  =fd ,  o2  o3  b4] 1  and  ii  =[oj  b2  «3  u,,]1  are 
the  joint  angles,  joint  velocities  and  joint  accelerations. 
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respectively.  Terms  on  the  left  side  of  the  equation  define 
the  torques  acting  on  the  system:  (7(n)  is  the  four 
dimensional  vector  of  torques  defining  the  effect  of  gravity 
at  the  joints,  /,(<>, 6, t)  is  ;t  4x1  vector  of  torques  defining  the 
force  applied  by  the  robot  to  the  hand  to  keep  it  on  the 
predefined  trajectory,  and  /*’(»,<'./)  is  the  4x  1  vector  defining 
all  other  torques  acting  at  the  joints.  The  last  term  includes 
the  friction  and  elastic  torques  resulting  from  the 
viscoelastic  properties  of  muscle,  as  well  as  the  active 
torques  (i.e.  those  due  to  reflex  and  voluntary  muscle 
contractions).  This  term  must  he  specified  to  test  specific 
hypotheses  about  how  the  arm  is  controlled  during  the 
locating  phase.  The  terms  on  the  right  side  of  the  equation 
describe  the  torques  related  to  the  motion  of  the  system: 
M( o)  is  the  4x4  inertia  tensor  (or  mass  matrix),  /7(o)  is  the 
4x6  matrix  of  Coriolis  coefficients  (mi  =  |«|U2  v?  'V'j 
Mu  0,6.,  I').  C(o)  is  the  4x4  matrix  of  centripetal  coefficients 
(,i’  =  Id,2  i)2*  A,2  (i/l').  The  Coriolis  and  centripetal  terms 
must  he  added  to  the  equation  because  the  frames  of 
reference  associated  with  the  coordinates  (joint  angles)  used 
in  the  model  are  not  Newtonian.  Explicit  expressions  for 
these  matrices  can  he  computed  with  standard  methods  |  6J. 
Because  they  tire  cumbersome,  wc  do  not  report  them  here, 
but  they  may  be  obtained  from  the  author  upon  request. 

The  joint  angles  0  must  satisfy  the  functional  endpoint 
constraint: 

Aotm  -  no  -  o  (2) 


where  /’(Of/))  is  the  hand  position  us  a  function  of  the  joint 
angles  and  /(/)  is  the  hand  position  imposed  by  the  robot. 
Because  the  constraint  holds  for  all  values  of  7,  it  holds  also 


for  all  derivatives  of  'Y(t).  The  first  time  derivative  of  the 
constraint  is: 


dt 


on  o)( 

c:>(> 


-  •/■</) 


(3) 


-  .7(0)0  -  i(D  -  0 


where  J(o)  is  the  3  by  4  Jacobian  matrix: 


•7(0) 


(4) 


Computing  the  second  derivative  of  the  constraint  yields: 


—{/’«))  rt/)}  -  T(D 


ill 


ill  dt) 


Cy’r 0)  ^  +  djWB  j f )  (p |  _  f{l) 


(5) 


do 


dO  dt). 

I  .1 


.7(0)0  ■<  77(0,0)  /(/)  -  0 


where 


77(0,0)  -- 


077,(0)0 

o'/y.imo 


(0) 


[077/0)0j 

and  H  (o'),  11,(0)  and  11,(0)  arc  the  Hessian  matrices: 


77,(0) 


flt\  ?2I\ 

,-o/n,  "  ['fl,ao4 

ah1,  fit\ 

argan,  "  i'OjiX/j 


(7) 


The  equation  of  motion  together  with  the  above 
constraint  defines  a  differential-algebraic  problem  [7).  This 
problem  can  be  solved  by  using  the  constraint  to  calculate 
the  value  of  the  unknown  term  T(o,6,t)  in  the  equation  of 
motion.  First,  the  equation  of  motion  is  rewritten  as: 

A7( 0 ) 0  -  (7(0,6)  +  /, (0,6.7)  (8) 


where  Q(a ,6)  is  the  four  dimensional  vector: 

77(0.0)  --/7«))|00|  -  C(())f02J  +  6(0)  +  /’((>.  6, 7)  (9) 

and  / ,(".6,7)  is  the  vector  of  torques  defined  by: 

/.(0.0.7)  =  J  '(())  7,(0, 6./)  ( 10) 

where  f(o)  is  die  transpose  of  the  Jacobian  matrix  and 
),(n.ii,/)  is  a  three  dimensional  vector  representing  the 
unknown  force  that  the  robot  must  exert  to  maintain  the  hand 
on  the  predetermined  trajectory  (sec  later).  Then,  the  value 
of  X(u.6,r)  can  be  found  by  solving  equation  (8)  for  o: 

0  -  M  •'((» {(7(0.6) '  .7  y(0)  2.(0, 0./)}  (11) 

Substituting  this  expression  in  equation  (5)  (i.e.,  the  second 
time  derivative  of  the  constraint)  yields: 

J(0)M  '(0){c(0,6)  ./  '(0)?u.0,0./)}  +  //(0,6)  -  7(0  0  (12) 

11'  the  3  by  3  matrix  ./(n)Af '(oj/fo)  is  invertible,  A.(«,6,/)  can 
be  computed: 

).,((). (’),/)  --■■■  [./(() )M  1  (()),/  '(OjJ'1 

{-./(() )A/  '(OJCXO.O)  77(0,6)  r  Ao} 

and  substituted  inlo  equtiUon  (10)  to  yield  an  ordinary 
second  order  differential  equation: 

6  =  M  '(0){67((l,  6)  './7(0)[,/(0)A7  '(())./  r(0)l  1  4 

-./(Oj/W  hJhCiO, 6)  //((),())  a  7’} 

Numerical  integral  ion.  To  integrate  equation  (14),  we 
used  an  algorithm  described  in  [  8  j  which  is  an 
implementation  with  step  size  control  and  dense  output  of  an 
explicit  Rungc  Kutla  method  of  order  5.  Because  rounding 
errors  may  lead  the  solution  to  diverge  from  the  constraint, 
we  also  followed  a  stabilizing  strategy  which  consisted  in 
projecting  the  joint  tingles  arid  velocities  obtained  at  each 
integration  step  onto  the  suhspacc  defined  by  the  constraint 

|7|." 

Let  us  assume  that  the  joint  angles  00  at  the  end  of  an 
integration  step  do  not  satisfy  the  constraint.  The  problem  is 
to  find  another  set  of  joint  angles  0  that  minimizes  the 
distance: 

T|iA'/(0h)(()  O0)!l2  (15) 

(the  matrix  M(0O)  is  not  necessary  hut.  it  modifies  the  norm 
in  a  way  that  makes  sense  in  this  context)  under  the  same 
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endpoint  constraint: 

P(0(O)  -  T{t)  -  0  (16) 

By  differentiating  equation  (15)  with  respect  to  0  and 
using  the  method  of  Lagrange's  multipliers,  we  obtain  the 
following  F(0,(i)  system  of  equations: 


F(0,g)  = 


M(0o)(O  -  00)  -  2  r(0Q)  fj. 

P(Q)  -  m 


0 

0 

(17) 


Because  the  forward  kinematics  operator  P(9)  is  nonlinear, 
the  system  cannot  be  solved  directly.  The  solution  can  be 
found  by  using  Newtons's  iterative  method,  by  repeati  ng  the 
following  two  steps.  In  the  first  step,  the  following  linear 
system  is  solved  using  a  standard  method  (e.g.  LU 
decomposition  with  back-substitution): 


M(%) 

2  7(60) 

\iok 

dlh 

,  2(0) 

0 

dfj.,. 

-F(.Qk.nk)  (18) 


\ 

Am. 

dH 

where  VF(6k,ixk)  is  the  gradient  of  F{0,p).  In  the  second 
step,  the  new  values  for  the  joint  angles  and  Lagrange 


(19) 


If  the  initial  values  [dO(),d//0]  are  not  too  far  away  from  the 
solution,  the  method  converges  rapidly  (the  number  of 
iterations  was  set  to  ten  with  initial  values  [d0o,dg()]  = 
[0o,O]). 

The  joint  angle  velocities  are  dealt  with  in  a  similar 
fashion.  Let  9  be  the  new  joint  angles  computed  above,  and 
let  o0  be  the  joint  angle  velocities  obtained  at  the  end  of  an 
integration  step.  Then  the  problem  is  to  find  the  joint  angle 
velocities  that  minimize: 

l|;Af(0)(0-0o)!!2  (20) 

while  satisfying  the  first  time-derivative  of  the  constraint: 

.7(0)0  -  fit)  -  0  (21) 


The  following  system  is  obtained  by  differentiating 
equation  (20)  with  respect  to  o  and  using  again  Lagrange's 
multipliers  method: 

M(0)(0  -  0O) 

2(0)9  - 

M(0)  2r(0) 

2(0)  0 

This  system  is  linear  in  respect  with  e  and  /r:and  can  be 
easily  solved  by  standard  methods. 

The  interaction  force.  Finally,  we  show  that  the  three- 
dimensional  vector  X(o,oj)  has  a  straightforward 
interpretation  in  terms  of  the  interaction  force.  In  vectorial 
notation,  the  work  is  the  dot  product  between  the  force  and 
displacement  vectors.  Because  the  work  (energy)  required 
to  maintain  the  hand  on  the  trajectory  must  be  the  same 
when  computed  in  different  coordinate  systems  [6],  we 
have: 

F  TSx  =  tt<50  (24) 


<  2r(0)g 

m 


0 

M(9)0O  ' 

F 

m 

(22) 


(23) 


where  5x  is  an  infinitesimal  displacement  of  the  hand  in 
space,  rSe  is  the  corresponding  displacement  in  joint  space,  F 
is  the  force  applied  by  the  robot  to  the  hand,  and  i  =  L(o,e  J) 
=  JT(o)X(e,e,t)  is  the  torque  component  added  to  the  equation 
of  motion  to  account  for  the  endpoint  constraint  (equation 
10).  By  definition,  the  Jacobian  2  =  <5a7c50  relates  joint 
velocity  to  and  cartesian  velocity.  Thus,  substituting  Sx  =  J 
(50  and  x  =  J  7(9)a.(o,6 2)  in  equation  (24)  and  simplifying 
yields: 

F  =  2"7t  =  J~TL(Q,Q,t)  =  2'727L(0,0J)  =  X(0,0,t)  (25) 

where  J'r  is  the  pseudo-inverse  of  the  transpose  of  the 
Jacobian.  Thus,  >,(o,oi)  is  the  force  applied  by  the  robot  to 
the  hand  to  maintain  it  on  the  predefined  trajectory  and,  by 
Newton’s  Third  Law,  -X(o,o 2)  is  the  force  applied  by  the 
hand  to  the  extremity  of  the  robotic  arm. 

Experimental  procedure.  We  recorded  the  interaction 
force  for  ten  blindfolded  subjects  while  a  robot  (M  ANUTEC 
R15,  Siemens)  moved  their  hand  laterally,  from  the  right  to 
the  left  (Figure  1  .A).  These  data  were  recorded  as  a  part  of 
a  series  of  experiments  on  pointing  movements  toward 
kinesthetic  targets  [9],  Subjects  were  instructed  to  let  the 
robot  guide  their  hand  to  some  position  in  space  which  they 
should  memorize.The  velocity  profile  of  the  imposed  motion 
was  either  lognormal,  gaussian  or  inverted  lognormal 
(Figure  2. A).  All  movements  lasted  1.5  s  and  were  30-cm 
long.  The  interaction  force  was  recorded  by  a  six  degrees  of 
freedom  force  transducer  (Mini  40,  ATI)  mounted  below  the 
handle  grasped  by  the  subjects  at  the  extremity  of  the  high 
impedance  robotic  arm. 


3.  Results 

In  this  section,  we  briefly  show  the  results  of  some 
simulations  and  compare  them  to  experimental  data.  Figure 
2.B  shows  the  interaction  force  for  each  velocity  profile, 
averaged  across  participants.  The  magnitude  and  orientation 
of  the  interaction  force  varied  considerably  during  the 
movement.  For  the  lognormal  velocity  profile,  the  magnitude 
of  the  force  was  maximum  at  the  beginning  of  the  movement 
while  the  opposite  was  true  for  the  inverted  lognormal 
velocity  profile.  Force  variations  during  the  movement  are 
less  obvious  for  the  gaussian  velocity  profile,  partly  because 
their  magnitude  is  small.  The  shoulder  and  elbow  positions 
were  not  recorded.  Thus,  the  comparison  between  simulated 
behavior  and  data  will  concern  mainly  the  interaction  force. 

In  the  simulations,  the  length,  diameter  and  mass  of  the 
two  cylinders  modeling  the  upper-  and  forearm  were  30  cm, 
4cm,  1 .43  kg  and  34cm,  3  cm,  and  1.32  kg  respectively.  The 
initial  hand  position  was  40  cm  in  front  of  and  25  cm  below 
the  shoulder.  In  the  first  set  of  simulations,  we  assumed  no 
other  forces  than  those  directly  related  to  the  inertial 
properties  of  the  arm.  Thus,  we  set 

F(0,0,O  =  0  (26) 
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Figure  2.  A:  Velocity  profiles  of  the  imposed  motions.  B:  Interaction  forces  during  the  imposed  motions.  The  origins  of  the  short  lines 
indicate  the  position  of  the  hand  every  20  ms.  The  direction  and  length  of  these  lines  denote  the  orientation  and  magnitude  of  the  interaction 
force  at  the  same  instant  in  time.  C:  Simulation  of  the  arm  movement  and  interaction  force  in  the  gaussian  velocity  profile  condition.  Uft 
panel:  Elbow  position  relative  to  the  shoulder  along  the  vertical  axis  (thick  line )  and  hand  velocity  ( thin  line).  Right  Panel:  Predicted  ( thick 
line)  and  average  (thin  line)  interaction  force.  D:  Simulations  with  the  weight  of  the  arm  supported  by  the  deltoid  (same  format  as  2.C).  E: 
Predicted  interaction  force  for  the  three  velocity  profiles  (same  format  as  2.B).  /*:  Magnitude  of  the  predicted  (thick  line)  and  actual  (thin 
line)  interaction  force. 


in  Equation  (3).  The  left  panel  of  Figure  2.C  shows  the  time 
course  of  the  predicted  elbow  height  (thick  line),  and  of  the 
hand  velocity  (thin  line)  in  the  gaussian  velocity  profile 
condition.  The  right  panel  contrasts  the  predicted  (thick 
line)  and  actual  (thin  line)  magnitude  of  the  interaction 
force.  The  simulation  was  extended  to  2.5  s  to  visualize  the 
behavior  of  the  arm  after  the  end  of  the  movement.  Under 
the  sole  influence  of  gravity,  the  elbow  swung  like  a 
pendulum  even  after  the  end  of  the  movement  because  there 
was  no  damping  in  the  system.  The  oscillations  in  the 
interaction  force  reflect  the  swinging  motion  of  the  elbow. 

Because  this  behavior  is  totally  unrealistic,  we  assumed 
in  the  next  set  of  simulations  that  the  subjects  actively 
supported  the  weight  of  their  arm  to  counteract  the  effect  of 
gravity.  The  parametrization  of  the  joint  angles  in  our  model 
afforded  a  simple  way  of  adding  an  elastic  force  to  keep  the 


elbow  elevated: 

F(0,0,0  =  Aj, 


0 

ti/2-0, 

0 

0 


(27) 


where  Kg  =  1.565  Nm/rad.  The  value  of  Ks  did  not  change 
throughout  the  movement  and  was  adjusted  so  that  the 
elbow  would  not  depart  from  its  initial  position  if  the  hand 
remained  immobile  at  the  center  of  the  workspace.  This 
elastic  force  may  be  thought  of  as  a  very  simple  model  of 
the  deltoid  muscle.  Figure  2.D  shows  that  the  simulations 
were  much  more  satisfactory.  Although  some  residual 
oscillations  of  the  elbow  remained  because  the  system  was 
not  dampened,  the  predicted  force  matched  the  actual  force 
much  belter.  Figure  2.E  shows  that  the  predicted  interaction 
forces  for  the  three  velocity  profiles  are  remarkably  similar 
to  the  data  (compare  with  Figure  2.B).  Finally,  Figure  2.F 
compares  the  time  course  of  the  magnitude  of  the  predicted 


and  actual  interaction  force.  The  actual  vs.  predicted  peaks 
of  the  force  magnitude  were  5.4  vs. 6.1  N,  2.0  vs  2.4  N,  and 
7.2  vs.  10.2  N  for  the  lognormal,  gaussian  and  inverted 
lognormal  velocity  profiles  respectively.  For  both 
asymmetric  profiles,  the  timing  of  the  large  peak  of  force 
was  well  predicted  (the  coefficients  of  correlation  between 
the  time-courses  of  the  actual  and  predicted  force 
magnitudes  were  .87,  .77,  and  .92  for  the  lognormal, 
gaussian  and  inverted  lognormal  velocity  profile 
respectively). 

4.  Discussion 

In  summary,  we  presented  a  method  to  predict  the 
interaction  force  when  the  hand  is  constrained  to  follow  a 
predetermined  trajectory  with  a  specific  velocity  profile. 
This  mathematical  tool  was  used  to  analyze  the  actual 
dynamic  behavior  of  the  arm  when  the  hand  was  moved  by 
a  robot. 

We  were  able  to  reproduce  the  most  salient  features  of  the 
interaction  force  by  assuming  that  an  upward  elastic  torque 
supported  the  weight  of  the  arm  throughout  the  movement. 
Like  the  equilibrium  point  [10]  and  the  virtual  target 
hypotheses  [1 1,  12],  this  scheme  emphasizes  the  role  of  the 
muscles'  elastic  properties  in  addition  to  the  inertial 
properties  of  the  arm.  Because  the  definition  of  this  force 
makes  no  reference  to  either  the  final  or  current  position  of 
the  hand,  these  results  suggest  that  the  subjects  did  not  try  to 
anticipate  the  displacements  of  the  hand 

The  magnitude  of  the  interaction  force  was  smallest  when 
the  imposed  motion  most  resembled  natural  movements 
(gaussian  velocity  profile).  However,  such  an  observation 
cannot  be  interpreted  as  evidence  that  the  subjects  responded 
differently  to  natural  vs  unnatural  stimuli  since  the  predicted 
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Computing  and  machines  keep  getting  more  versatile, 
powerful  and  complex.  This  trend  opens  the  door  to  a  new 
level  of  interactivity  between  humans  and  computers.  New 
applications  bring  together  the  human  intelligence  and  the 
machine  ability  to  carry  complex  tasks.  The  benefits  of  such 
symbiosis  are  safer,  faster  and  more  productive  applications. 
However,  fluent  collaboration  between  man  and  machine 
require  new  tools  that  allow  for  a  wider  range  of  information 
to  be  exchanged.  This  encourages  the  development  of  force- 
feedback  devices,  which  exploit  the  often  under-estimated 
human  sense  of  touch.  This  paper  describes  an  overview’  of  the 
Delta  Haptic  Device  developed  at  the  EPFL,  which  offers  6 
active  degree-of-freedom  together  with  an  outstanding 
mechanical  behavior. 


INTRODUCTION 

The  increasing  need  for  enhanced  human-computer 
interaction  (HCI)  is  pushing  for  new  interfaces  that  allow 
humans  and  machines  to  exchange  a  wider  range  of 
information.  As  a  consequence,  applications  involving  new 
interaction  modalities  such  as  machine  vision  and  virtual 
reality  are  being  developed.  Among  these  new  interfaces, 
haptic  devices  are  promised  a  place  of  choice.  Not  only 
does  haptic  make  difficult  manipulation  tasks  possible  or 
easier,  it  also  opens  the  door  to  a  wide  range  of  new 
applications  in  the  fields  of  simulation  and  assistance  to 
human  operators. 

Numerous  applications  can  benefit  from  haptic 
technology,  ranging  from  teleoperation  [2], [13]  to 
nanomanipulation  [6],  as  well  as  medical  simulators  and 
surgical  aids  [1],  Moreover,  force-feedback  devices  are 
moving  to  the  consumer  market,  and  are  invading  the 
gaming  industry  as  well  as  unexpected  other  areas  such  as 
automobile  industry  (BMW’s  new  Z9  for  instance). 

The  Virtual  Reality  and  Active  Interfaces  (VRAI) 
Group  at  the  Swiss  Federal  Institute  of  Technology  (EPFL) 
is  working  on  innovative  haptic  solutions  combining  state- 
of-the-art  parallel  mechanics  with  dedicated  control 
electronics,  which  can  be  coupled  with  any  application 
through  a  simple  API. 


RELATED  RESEARCH 

A  significant  research  effort  has  been  conducted 
towards  the  use  of  haptics  in  HCI  ([5],  [8],  [11],  [13]),  or 
more  specifically  on  what  are  the  mechanical  and  control 
constraints  that  come  into  play  when  developing  haptic 


devices  ([7],  [9],  [10],  [11],  [14],  [15]).  A  non  exhaustive 

list  that  summarize  some  of  the  most  important  criterions  in 

haptics  is  given  below: 

•  the  mechanical  system  should  have  low  inertia,  high 
stiffness  with  low  friction  (force  threshold)  and  no 
backlash  (mechanically  induced  force  discontinuity). 

•  the  force  actuators  should  enable  back-drivability, 
offer  a  high  dynamic  range,  a  sufficient  maximum 
force,  a  sufficient  force  output  resolution  and  a 
sufficient  force  and  torque  precision. 

•  the  position  sensors  need  to  have  a  good  position 
sensing  resolution  (depending  on  control  loop  refresh 
rate). 

•  force  sensors  should  be  added  as  close  as  possible  to 
the  human  user.  When  well  tuned,  they  greatly 
improve  the  overall  system  performance. 

•  the  local  control  loop  must  have  a  high  frequency 
(typically  >  1kHz)  to  ensure  (1)  that  vibrations  are 
kept  under  a  sensable  level  and  (2)  that  control 
stability  is  achieved  at  the  desired  contact  stiffness. 

•  the  global  force  interaction  low-frequency  loop, 
typically  linked  to  a  graphical  environment,  needs  to 
be  as  high  as  possible. 


DEVICE  DESCRIPTION 

Our  innovative  high-end  force-feedback  system  called 
the  Delta  Haptic  Device  meets  the  high  standards  required 
for  industrial  applications  by  combining  high  strength,  high 
stiffness  and  high  sensitivity.  It  is  shown  on  Figure  1.  This 
system  is  based  on  the  patented  Delta  robotic  structure  [3], 
which  provides  three  translational  degree-of-freedom 
(DOFs).  A  dedicated  mechanical  wrist  plug-in  provides  3 
rotational  DOFs  (shown  in  figure  1)  and  is  based  on  the 
Paramat  structure  (figure  3).  All  of  the  DOFs  are  fully 
active  and  generate  forces  and  torques  that  are  way  beyond 
the  ability  of  currently  available  devices  on  the  market. 
Dedicated  electronics  and  signal  processing  provide  the 
high-quality  control  required  for  credible  force  rendering. 
Figure  2  shows  an  overview  of  the  control  system. 
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Figure  1 :  Delta  Haptic  Device 

The  overall  performance  of  such  a  haptic  system  is 
closely  related  to  how  convincingly  the  human  sense  of 
touch  can  be  tricked.  There  are  many  functional 
components  in  a  generic  haptic  device,  which  contribute  to 
the  overall  system  quality.  They  can  be  split  in  two  main 
groups:  the  mechanical  elements  as  shown  in  figure  2,  and 
the  control  elements,  shown  in  figure  4. 


Figure  2:  the  DELTA  structure 


The  design  of  our  6  DOFs  system  addresses  most  of 
the  issues  described  above  (see  “Related  Research”).  The 
system  specs  are  summarized  in  the  following  table: 

TABLE  I 

:  system  specs 

Property 

Value 

workspace 

cylinder  0360mm  x  200mm 
+/-  20°  for  each  rotation 

force 

25N 

torque 

0.2  Nm 

resolution 

<0.1  mm  (translation) 

<  0.04°  (rotation) 

The  controller  described  in  figure  4  is  responsible  for 
two  tasks:  communication  with  the  PC,  and  low-level 
safety.  Its  main  purpose  is  to  command  the  motor 
amplifiers  and  transmit  the  device  position  back  to  the  PC. 
However,  since  the  forces  generated  by  the  Maxon  motors 
can  be  significant,  low-level  security  features  have  been 
introduced,  namely: 

•  check  speed  velocity  of  the  nacelle.  If  the  speed  goes 
beyond  a  given  threshold,  the  controller  cuts  the 
power  to  the  device  motors  and  electromagnetic 
brakes  are  applied 

•  check  structure  position  limits.  If  the  nacelle  is  set  to 
an  extreme  position  nearing  the  structure  limits, 
power  to  the  motors  is  cut  to  avoid  shocks 


Get  Sensor 


PC  Workstation  Delta  Controler  Delta  Device 

Figure  4:  DHD  control  architecture  overview 


Figure  3  -  the  PARAMAT  structure 

The  mechanical  system  displays  very  interesting 
properties,  thanks  to  its  parallel  structure  and  robust 
components.  The  main  advantages  are: 

•  low  inertia  /  light  weight  of  the  mobile  parts 

•  high  stiffness 

•  high  force/torque 

•  large  workspace 

•  no  backlash 


FUTURE  WORK 

We  are  currently  working  on  integrating  a  force  sensor 
into  the  nacelle  of  the  Delta  Haptic  Device.  The 
consequences  of  this  modification  are: 

•  lower  friction/force 

•  lower  inertia  of  the  system 

Preliminary  results  show  a  dramatic  improvement  of  these 
characteristics. 

Our  research  effort  for  the  near  future  will  lead  to  the 
development  of  plug-in  haptic  tools  that  will  enhance  the 
application  range  of  the  device.  At  the  same  time,  the 
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haptic  device  will  be  integrated  in  a  generic  human- 
computer  interaction  solution  involving  virtual  reality  and 
environment  sensing  in  real-time.  Several  partnerships  are 
being  conducted,  and  a  spin-off  company 
(www.forcedimension.com)  is  currently  being  created  that 
will  deal  with  the  commercialization  of  the  Delta  Haptic 
Device. 


CONCLUSION 

We  have  developed  an  innovative  haptic  device  called 
the  Delta  Haptic  Device  that  combines  a  parallel  mechanic 
structure  with  dedicated  electronic  and  software.  Our 
device  has  6  degrees-of- freedom  and  its  technical 
specifications  are  beyond  currently  available  haptic 
devices,  in  particular  as  far  as  workspace,  force  and  torques 
are  concerned. 

Our  research  effort  for  the  near  future  will  lead  to  the 
development  of  plug-in  haptic  tools  that  will  enhance  the 
application  range  of  the  device.  At  the  same  time,  the 
haptic  device  will  be  integrated  in  a  generic  human- 
computer  interaction  solution  involving  virtual  reality  and 
environment  sensing  in  real-time. 

Several  partnerships  are  being  conducted,  and  a  spin¬ 
off  company  is  currently  being  created  that  will  deal  with 
the  commercialization  of  the  Delta  Haptic  Device. 
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Abstract 

This  paper  demonstrates  that  the  perceived 
simultaneity  of  a  visual-haptic  stimulus  pair  is 
influenced  by  selective  attention.  The  results  provide 
an  explanation  for  the  individual  differences  in  the 
perceived  simultaneity,  as  found  in  a  previous  study. 


1.  Introduction 

Multimodal  information  is  of  great  advantage  in  the 
daily  perception  and  manipulation  of  our  environment 
compared  to  information  obtain  through  a  single 
sensory  modality.  Because  of  this,  it  is  often  desirable 
to  use  multimodal  displays  in  man-machine 
interaction.  To  benefit  from  multimodal  displays,  users 
must  be  able  to  experience  a  coherent  perception  of  the 
(virtual)  environment  by  integrating  input  from 
multiple  modalities.  One  perceptual  attribute  that 
provides  an  important  basis  for  intersensory  integration 
is  temporal  synchrony.  Synchronisation  is,  however,  a 
well-known  problem  in  multimodal  interfaces.  Due  to 
physical  and  technical  constraints,  such  as  computer 
processing  time,  interface  signals  are  often  delayed 
with  respect  to  each  other  and/or  the  action  of  the  user. 
Asynchronous  feedback  can  seriously  disrupt  many 
aspects  of  virtual  environment  simulations,  e.g.  it 
impedes  the  completion  time  of  manipulation  tasks 
(Ferrell,  1966).  Knowledge  about  temporal  sensitivities 
of  the  human  perceptual  system  is  therefore  essential 
in  the  design  of  man-machine  interfaces. 

This  research  focuses  on  the  sensitivity  of  human 
observers  to  time  delays  between  visual  and  haptic 
stimuli.  In  a  previous  study  (Vogels,  2001)  we 
measured  how  large  the  temporal  delay  between  visual 


and  haptic  stimuli  may  be  before  participants  start 
noticing  that  the  stimuli  are  asynchronous.  Participants 
moved  a  force-feedback  joystick  such  that  a  graphical 
object  on  a  monitor  would  hit  a  virtual  wall.  The 
collision  of  the  object  with  the  wall  was  felt  through 
the  joystick,  which  generated  a  counter  force  slightly 
before,  after  or  at  the  moment  of  collision.  The 
maximum  visual-haptic  delay  that  participants 
tolerated  was  on  average  45  ms.  The  range  in  which 
stimuli  were  judged  to  be  synchronous  was  centered 
around  a  visual  delay  of  about  7  ms.  However,  this  so 
called  point  of  subjective  simultaneity  (PSS)  was  liable 
to  individual  differences. 

It  was  suggested  that  one  of  the  factors  that  might 
play  a  role  in  the  variability  of  the  PSS  is  the  way  in 
which  participants  divide  their  attention  between  the 
two  modalities.  Researchers  have  shown  that 
manipulating  the  attention  of  the  participant  can 
influence  the  perception  of  simultaneity.  The  stimulus 
cued  by  the  experimenter  is  usually  perceived  earlier. 
This  effect  has  been  shown  for  stimuli  presented  to  the 
auditory  and  tactile  modalities  (Stone,  1926),  stimuli 
presented  within  the  auditory  modality  (Needham, 
1936)  and  stimuli  presented  within  the  visual  modality 
(e.g.  Stelmach  and  Herdman,  1991).  However,  some 
studies  failed  to  find  an  effect  of  attention  (Cairney, 
1975;  Jaskowski,  1993)  and  claimed  that  the  shift  in 
the  PSS  was  an  effect  of  response  bias  rather  than 
attention.  Spence  and  Driver  (1997)  showed  some 
appropriate  methods  for  distinguishing  attention  from 
other  confounding  factors. 

In  this  research  the  influence  of  attention  to  one 
modality  on  the  perceived  simultaneity  of  visual-haptic 
stimuli  was  studied.  In  order  to  assess  whether 
participants  were  indeed  directing  their  attention  to  the 
cued  modality,  we  used  a  reaction  time  procedure.  On 
each  trial  either  a  visual  stimulus,  a  haptic  stimulus  or 


167 


Eurohaptics  2001  Birmingham  July  2001 


an  asynchronous  visual-haptic  stimulus  pair  could  be 
presented.  When  one  stimulus  was  presented 
participants  were  required  to  make  a  discrimination 
response  regarding  the  modality  of  the  stimulus.  When 
both  stimuli  were  presented  participants  were  required 
to  make  a  temporal  order  judgement.  Attention  was 
manipulated  on  each  trial  by  presenting  a  cue  that 
predicted  which  modality  would  be  presented  or  which 
modality  would  be  presented  first.  To  ascertain  that  a 
possible  effect  of  cue  on  temporal  order  judgement 
was  due  to  attention  and  not  to  a  criterion  shift,  we 
tested  whether  participants  reacted  faster  on  validly 
cued  trials  than  on  invalid  trials  without  being  less 
accurate. 


2.  The  experiment 

This  experiment  investigated  the  influence  of  selective 
attention  to  one  modality  on  the  perception  of  visual- 
haptic  asynchronies. 

2.1.  Participants 

Participants  were  eight  students,  which  were  being 
paid  for  their  participation.  The  mean  age  was  20 
years.  All  participants  were  right  handed. 

2.2  Procedure 

Participants  were  seated  behind  a  computer  screen  and 
held  a  force-feedback  joystick  (Sidewinder  Force 
Feedback  Pro)  in  their  right  hand.  Participants  were 
instructed  to  focus  on  a  fixation  cross  on  the  middle  of 
the  screen.  At  the  beginning  of  a  trial  an  arrow  was 
displayed  for  2.0  s.  After  an  empty  time  interval  of  1.0 
to  1.5  s  a  stimulus  or  stimulus  pair  was  presented.  The 
stimulus  could  be  a  black  square  of  20  pixels  wide  on 
the  middle  of  the  screen  or  a  counter  force  of  5.5  N 
generated  by  the  joystick.  When  both  stimuli  were 
presented  they  were  separated  by  a  delay  that  ranged 
from  -240  ms  to  240  ms.  The  delay  was  defined  to  be 
negative  when  the  visual  stimulus  was  delayed  and 
positive  when  the  haptic  stimulus  was  delayed. 
Participants  were  asked  to  respond  as  soon  as  possible 
by  pressing  a  key  that  corresponded  to  the  modality 
that  was  presented  or  the  modality  that  was  presented 
first. 

The  arrow  at  the  beginning  of  the  trial  predicted 
which  modality  would  be  presented  or  which  modality 
would  be  presented  first.  An  upward  arrow  pointing  to 
the  fixation  cross  corresponded  to  the  visual  modality 
and  an  arrow  pointing  to  the  right,  where  the  joystick 
was  located,  corresponded  to  the  haptic  modality. 


Participants  were  instructed  to  direct  their  attention  to 
the  cued  modality. 

The  experiment  actually  consisted  of  two  types  of 
trials:  reaction  time  trials  (RT)  in  which  one  stimulus 
was  presented  and  temporal  order  trials  (TOJ)  in  which 
two  stimuli  were  presented.  In  the  RT  condition  the 
two  within-subjects  factors  were  stimulus  modality 
(visual  or  haptic)  and  cue  validity  (valid  or  invalid). 
There  were  224  (70%)  valid  trials,  where  the  stimulus 
modality  was  correctly  predicted  by  the  cue,  and  96 
(30%)  invalid  trials.  Cue  validity  was  the  same  for 
both  modalities,  which  were  presented  equally  often. 
In  the  TOJ  condition  the  within-subjects  factor  was  cue 
modality.  There  were  160  trials  with  a  visual  cue  and 
160  trials  with  a  haptic  cue.  In  each  cue  condition  16 
different  delays  were  presented,  10  times  each. 

Participants  had  a  lot  of  practice  before  the 
experiment  started.  They  also  participated  in  a  control 
experiment  in  which  only  160  TOJ  trials  were 
presented  without  the  cue.  This  experiment  severed  as 
a  baseline  for  the  perception  of  temporal  order. 


3.  Results 

We  first  analyse  the  RT  data  to  determine  whether 
participants  were  able  to  focus  their  attention  to  the 
cued  modality.  Trials  on  which  an  incorrect  response 
occurred  were  discarded  from  the  analysis.  In  addition, 
trials  on  which  the  RT  differed  more  than  3  standard 
deviations  from  the  mean  RT  were  removed.  These 
criteria  removed  less  than  4%  of  the  data.  Mean  RTs 
are  shown  in  Table  1.  We  performed  an  ANOVA  on 
the  RT  with  stimulus  modality  and  cue  validity  as 
within-subjects  factors.  There  was  a  significant  main 
effect  of  stimulus  modality  (Fji7=  1 1 .47,  p=0.012),  with 
participants  responding  more  rapidly  to  the  haptic 
stimulus  (M=361  ms)  than  to  the  visual  stimulus 
(A/=414  ms).  The  effect  of  cue  validity  was  also 
significant  (Fii7=9.38,  p=0.018),  with  participants 
responding  more  rapidly  on  validly  cued  trials  (M=362 
ms)  than  on  invalid  trials  (M=413  ms).  The  interaction 
effect  was  not  significant.  When  the  data  for  each 
participant  was  analysed  separately,  two  participants 
did  not  reveal  an  effect  of  validity.  Therefore,  these 
participants  were  not  included  in  the  following  analysis 

Table  1:  Mean  reaction  time  (in  ms)  and  the  standard 
error  for  the  visual  stimulus  and  the  haptic  stimulus. 


Cue  validity 


Valid 

I  Invalid 

Target  modality 

Mean 

SE 

Mean 

SE 

Visual 

391 

52 

437 

36 

Haptic 

332 

39 

391 

33 
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The  TOJ  data  was  plotted  as  in  Figure  1  and  fitted 
with  a  psychometric  function  to  determine  the  offset  of 
the  curve,  i.e.  the  delay  at  which  the  percentage  ‘visual 
first’  and  ‘haptic  first’  are  equal.  This  delay  is  called 
‘point  of  subjective  simultaneity’  (PSS).  We  performed 
an  ANOVA  on  the  PSS  with  cue  modality  as  the 
within-subjects  factor.  The  effect  of  cue  modality  was 
highly  significant  (Fli5=47.7,  p<0.001).  The  mean  PSS 
was  -59  ms  for  the  visual  cue  and  52  nis  for  the  haptic 
cue.  We  also  compared  the  PSS  values  with  those 
obtained  in  the  control  experiment.  The  mean  PSS  in 
the  absence  of  a  cue  was  35  ms  and  differed 
significantly  from  the  PSS  with  a  visual  cue  (F)i5=29.5, 
p<0.003)  and  with  a  haptic  cue  (Fi,5=35.4.  p<0.002). 
The  PSS  values  for  each  participant  are  shown  in 
Figure  2. 


Figure  1:  The  TOJ  data  for  one  participant.  The 
proportion  responses  “visual  first”  is  plotted  against 
the  delay  between  the  visual  and  haptic  stimulus.  The 
modality  of  the  cue  was  visual  (triangles)  or  haptic 
(circles).  The  data  was  fitted  with  a  psychometric 
function  (dark  line). 

4.  Discussion 

The  results  of  this  experiment  clearly  show  that  the 
perceived  simultaneity  of  visual-haptic  stimuli  is 
influenced  by  selective  attention.  The  PSS  shifted 
towards  visual  delays  when  participants  directed  then- 
attention  to  the  visual  stimulus  and  towards  haptic 
delays  when  participants  directed  their  attention  to  the 
haptic  stimulus.  This  means  that  if  the  visual  and 
haptic  stimuli  were  presented  simultaneously, 
participants  would  perceive  the  stimulus  to  which  they 
were  attending  as  occurring  earlier. 


125- 

100- 


12  .  . r . . , - r . . . - . - - - 
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participant 

Figure  2:  The  PSS  for  each  participant  in  the  case  of  a 
visual  cue  (black),  no  cue  (grey)  or  a  haptic  cue 
(white). 

Because  the  visual  and  haptic  stimuli  were  not 
presented  at  the  same  spatial  location,  it  is  impossible 
to  determine  whether  the  effect  was  due  to  spatial 
attention  or  to  attention  to  one  modality.  The  RT  data, 
however,  demonstrated  that  the  shift  in  the  PSS  was 
really  an  effect  of  attention.  If,  for  instance, 
participants  had  lowered  their  criterion  for  the  cued 
modality,  they  would  have  made  more  erroneous 
responses  on  trial  with  an  invalid  cue. 

The  results  provide  an  explanation  for  the 
individual  differences  in  the  perceived  simultaneity,  as 
found  by  Vogels  (2001).  When  participants  do  not 
receive  any  instructions  about  their  attention,  they 
divide  their  attention  between  the  two  modalities  in 
their  own  preferred  way. 


■  visual  cue 

■  no  cue 

□  haptic  cue 
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Abstract 

Haptic  feedback  plays  an  important  role  in  the 
development  of  high  fidelity  virtual  training 
environments  for  surgeons.  The  coupling  of  virtual 
simulation  environments  with  instmmented  data 
collection  methods  would  allow  quantification  of 
perfomance.  This  work  demonstrates  the  use  of  such  a 
system  to  achieve  quantitative  evaluation  of  suturing 
perfoimance.  Force  application,  time  to  task 
completion,  length  and  straightness  of  the  suture  have 
been  measured.  The  paper  concludes  that  performance 
improved  over  trials  in  terms  of  time  to  complete  and 
accuracy  of  the  length  of  suturing.  Force  feedback 
affected  force  application,  task  completion  time  and  the 
straightness  of  suturing. 


1.  Introduction 

There  is  increasing  interest  in  the  development  of  virtual 
reality  (VR)  systems  for  the  training  of  surgeons.  VR 
offers  many  benefits  to  training.  These  include  a 
reduction  in  costs  and  associated  patient  risk,  flexible 
access  to  training  material  and  the  possibility  of  self- 
paced  learning  [1][2].  The  trainee  is  able  to  develop 
surgical  skills  to  a  proficient  level  before  transferring 
them  into  the  operating  theatre.  The  fidelity  of  the  VR 
simulation  is  crucial  to  ensure  that  skills  transfer 
effectively. 

Strong  sensory  feedback  is  an  important  requirement 
for  realistic  surgical  simulation  [3].  Current  systems  vary 
in  the  quality  of  the  visual  and  tactile  information 
provided  to  the  user.  This  is  due  to  the  many  technical 
challenges  to  ensure  accuracy  [4][5]. 


The  ability  to  identify  tissue  properties  and  apply  the 
appropriate  handling  technique  is  an  important  skill  for 
the  trainee  to  acquire  [6].  A  realistic  simulation  should 
provide  the  user  with  force  feedback  in  order  for  the 
surgeon  to  be  able  to  detect  tissue  components.  These 
forces  are  small  and  exact,  while  difficult  to  recreate  [3]. 

Similarly,  in  order  to  produce  virtual  tissue, 
knowledge  is  required  of  real  tissue  properties  and  their 
respective  handling  forces.  Methods  to  collect  force 
application  data  therefore  are  required.  [7], 

Sens  Abie’s  PHANToM  desktop  device  has  been 
incorporated  into  many  current  simulation  designs 
[8]  [9],  It  provides  the  user  with  force  feedback  when 
virtual  objects  are  manipulated. 

Using  the  PHANToM,  this  paper  aims  to  demonstrate 
the  effect  offeree  feedback  information  on  performing  a 
basic  surgical  task.  The  effect  of  feedback  is  then  related 
to  force  application  and  improvement  of  the  task. 

2.  Method 

2.1  Participants 

Twenty  students  took  part  in  the  experiment.  They  had 
an  average  age  of  twenty  -five  years  (range  eighteen  to 
forty-five).  They  were  all  right  hand  dominant.  Each 
undertook  a  total  of  ten  trials. 

2.2  Equipment 

For  the  experimental  setting,  the  SensAble  PHANToM 
desktop  unit  [10],  run  in  Windows  NT  4.0,  was  utilised 
in  conjunction  with  a  suturing  simulation  [11].  The 
device  provides  resistance  to  the  users  movements  with  3 
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degrees  of  freedom  force  feedback  and  6  degrees  of 
freedom  position  sensing. 


Figure  1:  The  display  of  the  suturing  simulation. 


The  suturing  simulation  consisted  of  a  virtual  wound, 
needle  holders  and  needle  and  thread.  Sutures  could  be 
placed  in  the  skin  excision.  The  tissues  deformed  in  a 
manner  similar  to  actual  tissue,  when  pressure  was 
applied.  Force  feedback  was  provided  to  the  user  through 
the  haptic  end-effector  of  the  SensAble  PHANToM 
device. 


Figure  2:  The  experimental  setting:  the  needle- 
holders  attached  to  the  SensAble  PHANToM 
device. 


A  pair  of  needle  holders  was  loosely  attached  to  the 
end-effector.  As  a  consequence,  they  moved  in 
correspondence  to  the  virtual  instruments.  The  needle 
holders  were  instrumented  with  strain  gauges  to  allow 
measurement  of  the  forces  applied  during  their 
manipulation.  This  data  was  fed  directly  into  Microsoft 
Excel,  to  be  further  formatted  for  statistical 
manipulation. 

2.3  Procedure 

The  task  was  demonstrated  and  explained  to  each  subject 
by  the  experimenter.  Each  participant  then  performed 
two  test  sutures  to  familiarise  themselves  with  the  task 
and  the  experimental  setting. 

The  participants  were  asked  to  form  one  suture  across 
the  skin  excision.  It  was  specified  that  the  point  of  needle 
insertion  should  be  approximately  2cm  off  the  edge  of 
the  excision.  This  point  was  marked  on  the  simulation. 
Once  the  suture  had  been  formed  the  thread  was  pulled 
tight  to  bring  the  skin  edges  together  with  minimum 
overlap  and  tissue  distortion.  The  target  requirement  was 
a  4  cm  straight  stitch. 

Data  collection  began  at  the  first  movement  of  the 
needle  by  the  participant.  Time  was  measured 
automatically.  The  Daz  Wizard  program  was  initiated 
concurrently  to  allow  the  force  measurement  from  the 
needle-holders.  After  each  trial  the  resulting  stitch  was 
projected  and  traced  onto  acetate. 

Each  participant  completed  10  trials.  In  each,  a  full 
suture  procedure  was  formed  in  a  clear  wound.  The 
experimental  time  allowed  for  each  trial  was  limited  to 
40  seconds.  The  participants  were  assigned  to  one  of  two 
conditions.  For  the  first  the  PFIANToM  device  provided 
force  feedback  to  the  user  and  for  the  second  force 
feedback  was  not  applied. 


3.  Results 

The  results  were  analysed  to  determine  the  effect  of  force 
feedback  on  task  performance.  The  effect  of  trial  was 
further  analysed  to  investigate  how  performance  changed 
over  time. 

3.1  Effect  of  trial 

A  paired  sample  t-test  revealed  that  over  the  trials  the 
time  to  complete  a  stitch  was  significantly  reduced 
(p=0.01).  There  was  a  significant  change  in  the  length  of 
the  stitches  (p=0.04).  The  other  measures  were 
unaffected  by  trial.  Detailed  measurements  for  trial  1  and 
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10,  in  relation  to  their  average  (overall),  can  be  seen  in 
table  1. 


Condition 

Trial 

Time  (s) 

Force  (N) 

Length  (cm) 

Straightness 

score 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Force 

Feedback 

provided 

1 

28.24 

12.87 

5.59 

2.36 

2.19 

1.49 

4.6 

1.96 

10 

10.89 

1.98 

5.76 

2.01 

2.92 

0.74 

5.4 

0.7 

Overall 

20.17 

12.48 

5.67 

2.14 

2.52 

1.19 

5 

1.49 

Force 

Feedback 

withheld 

1 

22.79 

9.74 

3.94 

1.58 

2.27 

1.39 

4.6 

1.84 

10 

18.98 

11.65 

4.38 

2.19 

3.39 

1.48 

5.7 

0.67 

Overall 

20.53 

10.49 

4.16 

1.87 

2.8 

1.47 

5.15 

1.46 

Table  1:  Effect  of  trial  and  condition. 


3.2  Effect  of  condition 

An  independent  samples  t-test  revealed  that  the 
presence  of  force  feedback  significantly  affected  the 
force  applied  during  suture  production  (p=0.021).  A 
greater  level  of  force  was  applied  when  the  feedback 
was  activated.  The  time  to  complete  the  stitch  was 
affected  by  the  presence  of  force  feedback  (p=0.048). 
Feedback  resulted  in  an  overall  reduction  of  stitch 
completion  time.  Using  the  Mann -Whitney  test, 
straightness  was  affected  by  feedback  (p-0.018).  With 
feedback  higher  scores  for  straightness  were  achieved. 


4.  Discussion  and  conclusions 

Performance  has  been  shown  to  improve  over  trials. 
This  is  shown  by  a  reduction  in  the  time  to  complete 
and  improved  accuracy  of  the  length  of  the  stitch. 

To  increase  the  realism  of  surgical  simulations, 
force  feedback  should  be  provided  during  manipulation 
of  virtual  tissue.  The  results  indicate  that  force 
application  increases  as  does  time  to  complete,  with  the 
provision  of  force  feedback.  This  would  be  expected  in 
order  to  overcome  the  haptic  effects.  The  affect  on  force 
application  demonstrates  the  importance  of  accurate 
haptic  feedback.  Virtual  tissues  should  respond 
appropriately  to  allow  anticipation  of  real  tissue 
behaviour  and  for  skills  transfer  to  take  place.  Future 
work  should  demonstrate  how  force  application  is 
affected  by  variations  in  the  level  of  the  force  feedback. 

The  importance  of  collecting  force  application  data 
for  task  performance  has  been  demonstrated.  This 
information  has  been  derived  through  instrumented 
collection  of  data  in  a  simulation  environment.  The 
advantage  of  an  external  method  of  instrumented  data 
collection  is  its  transferability  between  the  operating 


theatre  and  simulated  models.  Determining  the 
appropriate  levels  of  force  application  for  particular 
tissues  provides  a  means  of  assessing  the  validity  of  a 
simulation,  enables  provision  of  feedback  during 
training  and  supports  the  establishment  of  benchmark 
levels  for  assessment. 
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Abstract 

This  paper  describes  a  pilot  experiment  which  examines 
the  effectiveness  of  using  force-feedback  technology  to 
assist  motion-impaired  computer  users  in  target  selection 
tasks.  Two  measures  to  evaluate  the  effectiveness  of  force- 
feedback  assistance  are  proposed.  The  radius  of  curvature 
of  the  cursor  path  and  the  cursor  speed  along  the  path  can 
capture  differences  in  cursor  movements  that  might  be  lost 
in  a  single  measure  such  as  time  to  target.  Experimental 
results  indicate  that  force-feedback  implemented  as  non- 
directional  viscous  damping  has  a  beneficial  effect  for 
some,  improving  times  to  target  by  up  to  50%.  Future 
work  includes  developing  other  forms  of  haptic  assistance, 
developing  new  measures  for  evaluation,  and  investigating 
the  implications  of  kinaesthesic  impairment. 


1.  Introduction 

People  with  motion  impairments  have  the  same  desire 
to  use  computers  as  able-bodied  people  [1],  but  the  typical 
keyboard  and  mouse  arrangement  for  computer  interaction 
often  presents  difficulties.  Consequently,  there  is  a  need 
to  improve  computer  access,  either  by  using  entirely  new 
modes  of  input  or  by  augmenting  existing  ones.  Focusing 
in  the  latter  area,  this  paper  describes  a  pilot  experiment 
that  examines  the  effectiveness  of  using  haptic  feedback  to 
enhance  human-computer  interaction.  In  particular,  this 
work  investigates  the  use  of  a  force-feedback  pointing 
device  as  a  means  of  assisting  motion-impaired  users  in 
target  selection  tasks,  and  extends  the  work  reported  in  [3]. 

To  determine  the  effectiveness  of  various  forms  of 
haptic  assistance,  methods  of  performance  evaluation  are 
required.  In  other  work,  measures  have  been  developed 
for  the  evaluation  of  different  pointing  devices  [4],  For 
the  current  study,  where  the  pointing  device  remains 
constant  hut  the  form  of  haptic  assistance  changes,  these 
measures  can  also  be  applied.  One  of  the  most  common  is 
movement  time.  This  method  of  evaluation,  however,  is 
based  on  a  single  measurement  per  trial,  and  although  it 
may  be  able  to  establish  that  a  difference  exists. 


determining  the  cause  of  the  difference  is  more  likely  to  be 
accomplished  through  analysis  of  cursor  movement 
throughout  the  trial  [4].  This  paper  describes  two 
measures  for  characterising  movement  throughout  a  trial  - 
the  cursor  velocity  along  the  cursor  path  and  the  radius  of 
curvature  of  the  cursor  path. 

2.  Method 

A  pilot  study  was  performed  to  investigate  the 
effectiveness  of  force-feedback  technology  as  a  means  of 
assisting  motion-impaired  users  in  target  selection  tasks. 
Four  volunteers  were  contacted  through  the  Papworth 
Trust,  a  charitable  organisation  dedicated  to  the  care  of 
motion-impaired  people.  A  description  of  each  user  is 
given  in  Table  1.  All  participants  were  familiar  with 
computer  and  mouse  use,  as  well  as  with  the  task  to  be 
completed.  Force  feedback  was  provided  with  a  Logitech 
Wingman  mouse. 


Table  1.  Computer  users  from  the  Papworth  Trust 


User 

Description 

PI3 

Athetoid  CP,  spasm,  wheelchair  user 

PI5 

Athetoid  CP,  deaf,  non-speaking,  ambulant 

P16 

Athetoid  CP,  ambulant 

PI7 

Friedrich’s  Ataxia,  tremor,  wheelchair  user 

For  the  task,  participants  were  presented  with  16 
“target”  circles  arranged  in  a  larger  circle  around  a  central 
“home”  circle,  as  shown  in  Figure  1.  The  computer  users 
were  required  to  select  each  target  in  a  random  order 
determined  by  the  software.  The  circles  became  red  to 
indicate  the  active  target.  Selection  was  achieved  by 
dwelling  inside  the  target  for  250ms.  After  successfully 
dwelling  on  an  active  target,  participants  had  to  select  the 
home  circle  before  the  next  target  was  activated. 

This  task  was  performed  under  4  damping  conditions  - 
None,  Acceleration  Damping,  Velocity  Damping,  and 
Combined  Damping.  In  None,  no  force-feedback  was 
applied  and  the  Wingman  operated  as  a  normal  mouse. 
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Figure  1.  Cursor  paths  for  PI7 

Non-directional  damping  was  implemented  as  a  viscous 
force  that  increased  linearly  with  mouse  acceleration 
( Acceleration  Damping),  mouse  velocity  (Velocity 
Damping),  and  a  combination  of  the  two  ( Combined 
Damping ).  In  Combined  Damping,  tire  velocity  damping 
was  at  half  the  strength  of  the  acceleration  damping. 

Each  cursor  path  was  recorded  by  logging  the  cursor 
position  and  system  time  each  lime  a  mouse  message  was 
processed  by  the  operating  system.  Consequently,  cursor 
position  was  sampled  at  irregular  time  intervals,  but  at  an 
average  rate  of  approximately  40  samples  per  second. 

3.  Performance  Measures 

In  work  that  has  been  ongoing  in  the  characterisation  of 
cursor  paths,  several  different  measures  have  been 
investigated.  Here,  two  of  the  more  intuitive  measures  - 
the  instantaneous  cursor  speed  along  the  path  and  the 
instantaneous  radius  of  curvature  of  the  path  -  arc 
described,  and  their  abilities  to  elicit  differences  in  the 
cursor  paths  are  illustrated. 

3.1.  Cursor  Speed 

Instantaneous  cursor  speed  is  approximated  as  the 
Euclidean  distance  between  two  sample  points  divided  by 
the  time  elapsed  between  them.  Cursor  speed  is  a  measure 
that  can  be  particularly  sensitive  to  detecting  the 
occurrence  of  spasm.  During  some  large  spasms,  high 
cursor  speeds  have  been  observed,  ranging  from  3 
pixels/ms  to  8  pixels/ms.  This  is  in  contrast  to  the  speeds 
observed  in  controlled  movements  by  able-bodied 
individuals,  typically  less  than  2  pixels/ms. 

Although  high  cursor  speeds  cannot  always  be 
attributed  to  spasm  with  absolute  certainty,  it  is  reasonable 
to  relate  a  high  cursor  speed  with  an  uncontrolled,  and 
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Figure  2.  Plots  of  the  reciprocal  of  radius  of 
curvature  for  the  cursor  paths  shown  in  Figure  1 

therefore  undesirable,  movement.  Observations  of  the 
speeds  of  able-bodied  and  motion-impaired  users  suggest 
that  a  threshold  of  3  pixels/ms  can  capture  the  difference 
between  controlled  and  uncontrolled  movement.  The 
number  of  times  the  cursor  speed  exceeds  this  threshold 
can  be  used  to  measure  the  frequency  at  which 
uncontrolled  movements  occur.  It  will  later  be 
demonstrated  that  damping  as  a  form  of  haptic  assistance 
can  help  reduce  the  frequency  of  occurrence  of  high¬ 
speed  uncontrolled  movements. 

3.2.  Radius  of  Curvature 

A  curved  path  can  be  represented  geometrically  as  a 
series  of  differential  arc  segments,  each  formed  from  the 
arc  of  an  associated  circle  having  a  particular  radius  of 
curvature  [2] .  The  radius  of  curvature,  calculated  here  for 
an  arc  length  of  10  pixels,  can  provide  information  about 
the  number  of  “twists  and  turns”  in  the  path.  A  small 
radius  (e.g.  <10  pixels)  indicates  a  sharp  turn,  and  a  large 
one  (e.g.  >50  pixels)  indicates  relatively  straight  motion. 

Figure  1  shows  the  cursor  paths  for  three  targets.  All 
three  paths  have  similar  time  to  target  values  {(a)  = 
3609ms,  (b)  =  3974ms,  (c)  =  4635ms),  and  so  an  analysis 
based  on  this  single  measurement  would  not  have  revealed 
any  differences  among  them.  A  look  at  the  cursor  paths, 
however,  shows  that  the  behaviours  are  quite  different. 
The  radius  of  curvature  plots  (Figure  2)  are  able  to  elicit 
these  differences.  The  values  are  plotted  as  the  reciprocal 
of  the  radius  of  curvature,  so  that  high  values  indicate 
highly  curved  movement  and  low  values  indicate 
relatively  straight  movement. 
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4.  Results 

Figure  3  shows  the  average  time  to  target  for  tire 
various  damping  conditions  (6  sequences  per  damping 
condition  per  participant,  16  targets  per  sequence).  It 
appears  that  all  forms  of  damping  have  a  favourable  effect 
for  PI3,  reducing  times  by  over  50%.  The  damping  does 
not  appear  to  have  a  great  effect  on  times  for  the  other 
participants.  It  may  be  that  the  nature  of  their  impairments 
does  not  benefit  so  explicitly  from  the  damping,  or  that  the 
benefit  is  balanced  by  the  extra  motor  load  produced  in 
moving  the  mouse.  This  will  be  studied  further. 

Given  the  reduction  in  time  to  target  for  PI3,  the  cursor 
speeds  were  studied  further.  Table  2  shows  the  number  of 
times  the  cursor  speed  exceeded  the  threshold  of  3 
pixels/ms  for  the  four  different  conditions.  The  various 
forms  of  damping  appear  to  reduce  the  frequency  of 
uncontrolled,  high-speed  movements  by  70-90%. 
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Figure  3.  Time  to  Target  vs.  Damping  Type 


5.  Conclusion 

A  pilot  study  has  been  performed  to  investigate  the 
effectiveness  of  force-feedback  technology  to  assist 
motion-impaired  users  in  target-selection  tasks.  Two 
measures  for  evaluating  the  effectiveness  of  haptic 
assistance  have  been  proposed.  The  radius  of  curvature  of 
the  cursor  path  and  the  cursor  speed  along  the  path  can 
capture  characteristics  of  the  movement  throughout  a  trial 
which  may  otherwise  be  lost  in  a  single  measure  such  as 
time  to  target. 

In  the  experiment,  haptic  assistance  was  implemented 
as  non-directional  viscous  damping.  The  force-feedback 
appears  to  be  beneficial  for  some,  reducing  the  time  to 
target  by  over  50%  for  one  user  with  spasm.  Since  force- 
feedback  must  be  applied  appropriately  to  be  of  benefit, 
future  work  includes  developing  and  evaluating  other 
forms  of  haptic  assistance. 

For  evaluation,  further  studies  will  be  conducted  to 
determine  the  reliability  of  measures  within  individuals, 
across  individuals,  and  across  tasks.  The  measures  will  be 
applied  to  evaluate  the  effect  of  various  forms  of  haptic 
assistance.  Further  study  of  the  cursor  paths  will  be 
conducted  to  gain  insight  that  may  lead  to  the  development 
of  more  appropriate  forms  of  haptic  assistance.  Finally, 
the  implications  of  kinaesthesic  impairment  on  haptic 
assistance  remain  to  be  investigated. 
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Table  2.  Thresholding  of  cursor  speed  for  PI3 


Condition 

No.  of  times 
above  3  pixels/ms 

None 

70 

Acceleration  Damping 

20 

Velocity  Damping 

6 

Combined  Damping 

14 
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Abstract 

We  investigated  how  various  forms  of  prior  knowledge 
influence  learning  speed  and  recognition  performance  of 
previously  unfamiliar  three-dimensional  (3D)  objects.  The 
learning  objects  were  three  ’molecule’  models  each  of  which 
was  composed  of  four  spheres.  Prior  knowledge  was  varied 
in  terms  of  the  sensory  modality  ( visual  versus  haptic  versus 
visuohaptic)  employed  during  a  so-called  exploration 
phase.  Following  this  exploration  subjects  were  trained  in  a. 
visual  learning  task  with  a  fixed  set  of  two-dimensional 
(2D)  views  of  the  learning  objects.  We  found  a  significant 
effect  of  sensory  modality  employed  during  the  exploration 
phase  on  both  learning  rate  and  recognition  performance  in 
the  subsequent  visual  learning  task.  Specifically,  a  short 
period  of  haptic  or  visuohaptic  exploration  reduced 
learning  duration  by  about  60%  relative  to  conditions  with 
either  none  or  visual-only  exploration.  Computer 
simulations  on  the  basis  of  the  behavioural  data  suggest 
that  prior  haptic  exploration  stimulates  the  evolution  of 
object  representations  which  are  characterized  by  an 
increased  differentiation  between  attribute  values  and  a 
pronounced  structural  encoding. 

1.  Introduction 

Concerning  the  quality  of  internal  representations  underlying 
human  object  recognition  there  are  two  dominant  views.  On 
the  one  hand,  it  has  been  postulated  that  objects  are  mentally 
represented  by  three-dimensional  (3D),  object  centred,  part- 
based  descriptions  [1,2,3].  On  the  other  hand,  more  recent 
studies  have  provided  evidence  for  the  notion  that  3D 
objects  are  in  fact  represented  in  terms  of  multiple,  viewer 
centred,  two-dimensional  (2D)  views,  among  which  the 
visual  system  interpolates  if  necessary  [4,5,6]. 

Experimentally,  both  hypotheses  have  been  mainly  tested 
in  studies  where  the  test  object  was  presented  from  di  fferent 
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perspectives  and  where  the  change  of  the  error  rate  or 
response  latency  has  been  measured  in  identification  tasks  as 
a  function  of  viewing  angle.  However,  it  has  been  shown 
that  the  two  alternative  explanations  are  not  readily 
distinguished  within  this  paradigm.  First,  the  dependency 
from  viewpoint  is  itself  dependent  on  object  familiarity  [7], 
demonstrating  the  necessity  to  take  into  account  learning 
processes.  Second,  a  closer  inspection  of  the  apparent 
complementary  approaches  shows  that  the  assumed 
vie  vvpoint-invariance  of  the  3D  hypothesis  holds  only  under 
certain  conditions  [8].  Conversely,  representations  in  terms 
of  multiple  2D  views  may  become  quasi-independent  from 
viewpoint,  if  the  number  of  views  is  sufficiently  large,  or  if 
the  interpolation  mechanism  between  views  becomes  more 
efficient  due  to  training. 

A  further  complication  arises  from  the  fact  that  it  is  still 
unclear  to  what  extent  depth  information  is  used  for  building 
internal  representations  of  3D  shapes.  There  is  evidence  that 
the  influence  of  binocular  disparity  (shown  to  be  strongest 
contributor  to  depth  information  in  recognition)  declines 
with  increasing  familiarity  [9],  In  contrast,  experiments  in 
the  field  of  haptic  object  recognition  demonstrate  that  the 
identity  of  familiar  objects  may  be  established  very  quickly 
and  seems  to  be  mediated  mainly  by  3D  structural 
information  [10,11].  The  latter  result  would  suggest  a 
participation  of  polymodal  sensory  systems  in  the 
ontogenesis  of  mental  object  representations. 

In  this  study  we  have  investigated,  within  a  supervised 
learning  paradigm,  how  prior  knowledge  of  objects  in 
various  sensory  modalities  influences  learning  speed, 
recognition  performance  and  the  ability  of  spatial 
generalization.  The  learning  objects  were  molecule-like 
models  made  up  of  four  spheres.  Prior  knowledge  was 
varied  in  terms  of  the  sensory  modality  (visual  versus  haptic 
versus  visuohaptic)  employed  during  a  so-called  exploration 
phase.  Following  this  exploration  subjects  were  trained  in  a 
visual  learning  task  with  a  fixed  set  of  two-dimensional 
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(2D)  views  of  the  learning  objects.  Learning  speed  was 
measured  as  the  number  of  training  cycles  that  were 
necessary  to  reach  a  given  criterion  concerning  the 
classification  of  the  learning  views.  In  a  subsequent 
generalization  test,  recognition  was  assessed  by  the  ability  to 
correctly  assign  a  set  of  novel  2D  views  of  the  previously 
learned  objects.  To  gain  further  insight  into  the  nature  of  the 
mental  object  representations  acquired  during  learning 
additional  computer  simulations  were  conducted.  The 
simulations  employed  a  recognition  model  based  on 
relational  evidence  theory  [12]  which  allows  to  characterize 
object  representations  in  terms  of  components  and  relational 
rules. 

2.  Method 

The  experiments  employed  a  set  of  three  objects.  Each 
object  was  composed  of  four  spheres,  with  three  of  them 
forming  an  isosceles  triangle  and  the  fourth  being  placed 
perpendicular  above  the  centre  of  one  of  the  base  spheres 
rendering  the  objects  similar  to  ’molecule’  models.  The 
objects  were  generated  both  as  virtual  models  and  as 
physical  models.  Virtual  models  were  constructed  and 
displayed  on  a  SGI  02  workstation  using  the  Open  Inventor 
software  package.  The  physical  models  were  constructed  of 
styrofoam  balls.  From  the  virtual  models,  2D  views  were 
generated  as  perspective  projections  of  the  objects  onto  the 
screen  plane  of  the  computer  display.  Two  sets  of  views 
were  generated,  a  training  set  comprising  22  images  and  a 
test  set  comprising  83  images.  The  two  sets  were  obtained 
by  sampling  the  viewing  sphere  in  60  deg  steps  (training  set) 
and  30  deg  steps  (test  set),  respectively,  and  by  eliminating 
all  views  that  were  redundant  due  to  object  symmetry.  At  the 
viewing  distance  of  lm  the  images  appeared  under  a  visual 
angle  of  1.5  deg. 

The  experiment  was  divided  into  three  parts,  a  so-called 
exploration  phase,  a  supervised-learning  phase  and 
generalisation  test.  During  the  exploration  phase  the 
participants  were  allowed  to  familiarize  themselves  with  the 
objects.  The  subjects  were  divided  into  three  groups  that 
differed  concerning  the  sensory  modality  employed  during 
the  exploration:  visual,  haptic,  visuohaptic.  Haptic  and 
visuohaptic  exploration  was  done  by  grasping  and 
manipulating  the  physical  object  models.  In  the  purely 
haptic  case  the  subjects  were  blindfolded.  Visual  exploration 
was  mediated  by  actively  rotating  the  virtual  object  models 
on  the  screen  by  means  of  the  computer  mouse.  In  each  case, 
the  exploration  phase  lasted  for  3  minutes  and  was  followed 
immediately  by  the  visucil-learning  phase.  In  addition,  there 
was  a  fourth  (control)  group  where  subjects  started 
immediately  with  the  visual  learning  without  any  prior 
exploration. 

The  supervised  learning  procedure  consisted  of 


subsequent  learning  units  (see  [13]).  In  each  unit  the  each 
view  of  the  learning  set  was  presented  three  times  in  random 
order.  The  exposure  duration  of  each  view  was  250  msec, 
and  each  presentation  was  followed  by  the  corresponding 
object  label  displayed  for  one  second.  The  learning  unit 
ended  with  a  recognition  test  to  assess  the  learning  status  of 
the  subject.  Here  each  view  of  the  learning  set  was  shown 
once  for  250  msec  and  labelled  by  the  observer.  The 
sequence  of  learning  units  continued  until  the  subject  had 
reached  the  learning  criterion  of  90%  correct  responses  in 
the  recognition  test.  After  having  completed  learning  the 
observers  continued  with  the  generalisation  test.  Here  all 
views  of  the  test  set  were  shown  in  random  order.  Each  test 
view  had  to  be  assigned  to  the  previously  learned  objects. 

3.  Results 

The  results  show  that  learning  duration,  measured  by  the 
number  of  learning  units  necessary  to  reach  the  criterion, 
was  significantly  affected  by  sensory  modality  employed 
during  the  exploration  phase.  Specifically,  the  short  haptic 
exploration  phase  reduced  the  learning  duration  by  about 
60%  as  compared  to  the  control  condition.  Haptic 
exploration  also  distinctly  improved  the  ability  to  recognize 
novel  views  of  the  previously  learned  objects  in  the 
generalisation  test,  from  a  level  of  about  60%  to  about  75% 
correct  responses.  The  advantage  in  learning  and 
generalisation  was  the  same  no  matter  whether  the  haptic 
exploration  was  accompanied  by  sight  (in  the  visuohaptic 
condition)  or  not  (in  the  haptic  condition).  However,  a 
purely  visual  exploration  of  the  objects  (visual  condition) 
yielded  no  significant  advantage  relative  to  the  control 
condition,  neither  in  terms  of  learning  speed  nor 
generalization. 

To  explore  how  prior  knowledge  acquired  during  the 
exploration  phase  specifically  affected  the  ontogenesis  of 
the  object  representations  during  the  subsequent  visual 
learning  we  conducted  computer  simulations  on  the  basis  of 
the  confusion  error  data.  The  simulations  employed 
CLARET,  a  recognition  model  based  on  relational 
evidence  theory  [12].  CLARET  involves  a  process  of  rule 
induction,  in  which  a  taxonomy  of  rules  is  created  to 
discriminate  between  objects.  The  rules  refer  to  relational 
attributes,  such  as  distance,  angle  or  size  ratio,  that  are  used 
to  describe  object  structure.  Starting  from  an  initial  rule, 
defined  by  an  attribute  range  which  is  satisfied  by  the 
relations  of  all  existing  objects,  rule  refinement  proceeds  in 
two  ways:  (1)  By  re-partitioning  the  attribute  space  via 
clustering,  thus  producing  rules  of  increasing  attribute 
specificity,  and  (2)  by  incorporating  relational  information 
in  terms  of  rules  which  include  other  rules  in  their  body. 
Thus  the  resulting  rule  hierarchy  is  characterized  by  two 
properties:  attribute  specificity  and  relational  depth, 
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Since  the  behavioural  data  revealed  a  distinct  dichotomy 
between  subjects  with  haptic  versus  those  with  non-haptic 
prior  object  experience,  the  data  of  the  subjects  were 
divided  into  two  groups,  haptic  and  non-haptic,  respectively. 
The  simulations  yielded  for  the  haptic  group  a  high  degree 
of  differentiation  between  values  along  the  feature 
dimensions,  with  an  optimal  fit  around  a  value  of  5 
partitions.  In  contrast,  observers  with  no  prior  experience 
tended  to  differentiate  between  attribute  values  much  more 
coarsely,  with  an  optimum  partition  number  around  1-2. 
Concerning  relational  depth,  only  the  fit  for  the  haptic  but 
not  for  the  non-haptic  group  improved  as  more  rules  were 
included  in  rule  generation.  This  warrants  the  conclusion 
that  object  rules  in  the  former  case  extend  over  multiple 
object  components  whereas  they  remain  confined  to  isolated 
component  pairs  in  the  latter. 

4.  Conclusion 

We  have  shown  that  a  prior  haptic  exploration  of  3D  objects 
leads  to  a  substantial  facilitation  in  visual  learning  and 
generalisation.  Computer  simulations  suggest  that  theses 
facilitation  may  be  due  to  a  different  representational  format 
of  the  underlying  mental  object  representations. 
Accordingly,  prior  object  knowledge  acquired  via  the  haptic 
sense  stimulates  the  development  of  representations  that  are 
characterized  by  a  high  degree  of  attribute  differentiation 
and  an  enhanced  use  of  relational  information.  The  present 
results  are  preliminary  in  the  sense  that  they  do  not  yet  allow 
conclusions  concerning  the  dimensionality  of  mental 
representations  of  3D  objects.  However,  the  observed 
dependency  of  visual  learning  on  haptic  information 
indicates  that  such  object  representations  may  have  an 
intrinsic polymodal  quality,  thus  questioning  the  validitiy  of 
purely  vision-based  accounts  of  object  recognition. 
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Abstract 

In  this  paper,  a  new  approach  for  haptic  rendering  of  high  resolution  voxel-based  anatomic  models  is  presented. 
For  visualization  the  surface  location  is  determined  by  a  ray-casting  algorithm  at  sub-voxel  resolution.  Since  the 
same  algorithm  is  used  for  the  haptics  as  well,  a  very  high  level  of  detail  is  achieved  for  haptic  feedback.  Both 
graphical  and  haptic  representation  is  congruent. 

The  interaction  forces  are  calculated  based  on  a  collision  detection  between  an  arbitrary  sized  sphere-shaped  tool 
and  an  arbitrary  complex  anatomic  model.  Forces  are  calculated  at  an  update  rate  of  6000  Hz  and  sent  to  a  3- 
Degree-of-Freedom  (3-DOF)  Phantom  device  [7].  Compared  to  point-based  haptic  rendering,  the  unique 
combination  of  the  sphere-based  approach  in  combination  with  sub- voxel  rendering  provides  more  realistic  and 
very  detailed  tactile  sensations. 

1 1ntroduction 

Today  haptic  rendering  is  mostly  based  on  traditional  computer  graphics  methods  where  objects  are  represented 
by  polygons  only.  However,  using  a  surface  based  representation  to  create  a  model  for  medical  applications,  the 
knowledge  about  the  interior  structures  of  the  organs  is  lost.  This  knowledge  is  very  important  when  simulating 
interactive  cutting  operations  such  as  required  for  surgery  simulations. 

In  addition  we  realized  that  today  3-DOF  haptic  rendering  is  mostly  point-based,  i.e.  only  one  point  is  used  to 
calculate  collisions  and  forces.  This  induces  several  problems: 

•  Discontinuities  (e.g.  sharp  edges)  on  the  surface  can  lead  to  discontinuities  in  the  haptic  display 

•  The  virtual  tool  can  reach  points  which  can  not  be  reached  by  the  simulated  real  world  tool 

Our  approach  uses  a  tool-based  collision  detection  which  was  inspired  by  [I].  For  collision  detection  and  force 
calculation  we  used  a  sphere-shaped  tool  which  simulates  a  drill  as  it  is  used  in  petrous  bone  surgery. 

1.1  Previous  work 

The  voxel-based  approach  to  haptic  rendering  presented  in  [1]  enables  6-DOF  manipulation  of  a  modestly  sized 
rigid  object  within  an  arbitrary  complex  environment  of  static  objects.  One  drawback  of  this  method  is  that  the 
haptic  rendering  is  at  voxel  level  accuracy  only.  Our  approach  differs  from  this  work  primarily  in  the  surface 
representation.  While  our  model  is  also  using  a  voxel  representation,  the  exact  location  of  the  surfaces  is 
calculated  by  a  ray-casting  algorithm  at  sub-voxel  resolution.  That  leads  to  smooth  surfaces  while  even  small 
surface  details  can  be  represented. 

2  Methods 

Mainly  the  following  three  aspects  are  involved  in  our  approach: 

1.  The  collision  force  on  the  surface  of  the  objects  can  be  computed  by  an  algorithm  which  was  inspired 
by  [1]. 

2.  To  calculate  correct  forces  for  deeper  tool-object  penetrations,  a  modified  proxy  algorithm  [4,5]  was 
used. 

3.  In  contrast  to  other  representations,  our  model  does  not  include  an  explicit  representation  of  the  surface. 
It  is  rather  calculated  by  a  ray-casting  algorithm  at  sub- voxel  resolution  [2].  Such  calculations  are 
computationally  expensive  and  thus  they  should  be  minimized  to  the  extend  absolutely  necessary. 
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2.1  Representation  of  the  tool 

The  tool  is  represented  by  a  number  of  points  which  are  distributed  at  preferably  equal  distances  over  the  tool 
surface,  together  with  the  inward  pointing  surface  normals  (Fig.  1).  In  our  case  all  inward  pointing  normal 
vectors  are  ending  at  the  center  point  of  the  sphere.  To  get  an  adequate  representation  of  the  shape  while 
reducing  computation  to  a  minimum,  we  used  26  sample  points  on  the  surface.  Depending  on  the  size  of  the 
sphere  more  point  samples  might  be  required. 


Fig.  1 :  Representation  of  the  tool  Fig.  2:  Calculation  of  collision  force 

2.2  Haptic  rendering  of  the  surface 

When  a  collision  occurs,  two  parameters  of  the  collision  force  must  be  calculated: 

•  Direction  and 

•  Magnitude  of  collision  force 

First  of  all,  the  surface  points  of  the  sphere  are  checked,  whether  they  are  inside  or  outside  the  object.  All  surface 
points  which  are  inside  the  volume  are  traced  in  direction  of  the  inward  pointing  normal  until  the  surface  is 
found  (Fig.  2)  or  the  sphere  center  point  is  reached.  All  found  vectors  are  added  and  the  direction  of  the  resulting 
vector  is  the  direction  of  the  force  vector  which  must  be  applied  to  the  haptic  device.  The  problem  of  calculating 
the  magnitude  of  the  force  vector  is  still  under  investigation;  so  far,  we  use  a  heuristic  method  which  averages 
the  magnitudes  of  the  surface  vectors.  This  leads  to  deeper  penetration  while  more  tool  points  are  in  contact  with 
an  object. 

2.3  Proxy  object  algorithm 

Any  haptic  device  has  a  limited  force  which  can  be  applied  to  it.  When  the  user  pushes  harder,  it  is  possible  that 
the  physical  position  of  the  tool  immerses  completely  into  the  virtual  model.  In  this  case  a  calculation  of  the 
force  direction  as  described  above  is  not  possible  anymore  because  the  sphere  center  point  is  in  the  object  itself. 
To  overcome  that  limitation  a  modified  proxy  object  [5,6]  was  implemented.  In  free  space,  the  position  of  the 
proxy  object  and  of  the  device  is  identical.  When  the  haptic  device  moves  into  an  object,  the  proxy  remains  on 
the  object’s  surface.  When  the  proxy  position  is  known,  the  resulting  force  vector  is  proportional  to  the 

difference  vector  between  the  proxy  and  the  device  position. 

In  order  to  update  the  proxy  position  while  the  device  is  moving,  the  distance  between  the  device  position  and 
the  proxy  must  be  locally  minimized  by  regarding  the  surface  constraints.  Since  searching  for  the  local  minimum 
would  be  computationally  very  expensive  in  our  model,  a  new  algorithm  was  implemented.  Whenever  more  than 
a  certain  number  of  surface  sample  points  are  in  contact  with  an  object,  the  way  back  to  the  proxy  is  traced  until 
the  number  of  contacts  is  below  the  limit  or  until  the  proxy  is  reached.  At  that  location  the  force  vector  is 
calculated  as  described  above  and  the  proxy  is  set  to  that  position. 

2.4  Implementation 

The  system  was  implemented  on  a  Compaq  SP750  workstation.  The  workstation  has  two  Pentium  III  Xeon 
processors  at  866  MHz  and  is  equipped  with  2GB  RAM.  As  haptic  device  we  used  a  3-DOF  Phantom  Premium 
1.0A  [7].  Our  system  was  running  under  SuSE  Linux  7.1.  For  connecting  the  device  directly  to  the  system,  we 
used  the  open-source  PHANToM  Linux-driver  from  [4]. 

The  haptic  process  runs  at  an  update  rate  of  6000Hz.  The  most  time  consuming  calculation  is  the  calculation  if  a 
point  is  in  contact  with  the  objects.  By  using  26  sample  points  on  the  sphere,  60  such  calculations  are  executed  at 
every  haptic  update  on  average. 
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3  Results 

With  our  approach  we  achieve  high  quality  haptic  rendering  of  arbitrary  complex  anatomic  models  [9]  as  shown 
in  figure  3.  The  perception  of  spatial  relationships  is  greatly  enhanced  with  haptic  feedback.  Even  very  small 
surface  details  or  small  objects  like  nerves  and  vessels  can  be  sensed  realistically  due  to  our  sub-voxel  based 
approach.  As  a  major  application  of  our  method  we  are  working  on  a  system  for  the  simulation  of  middle  ear 
surgery  (fig.  4).  Here,  the  haptic  device  is  used  to  simulate  drilling  into  the  mastoid  bone  [8].  Using  a  sphere- 
based  collision  detection  proved  to  be  much  more  realistic  than  using  point-based  haptic  interaction.  Drilled 
holes  and  tubes  in  the  bone  could  be  intuitively  probed  with  the  haptic  device. 

4  Conclusions  and  future  work 

The  approach  presented  here  overcomes  several  problems  of  point-based  haptic  rendering.  The  shape  based 
collision  detection  makes  interactions  more  realistic,  especially  for  the  use  in  surgery  simulations.  Another 
important  advantage  is  that  surface  details  can  be  sensed  as  expected  from  the  graphical  representation.  Sub¬ 
voxel  rendering  leads  to  both  a  graphical  and  haptical  realistic  and  high  detailed  display. 

While  the  current  implementation  is  limited  to  sphere-shaped  tools,  future  implementations  will  extend  this  to 
more  general  shapes.  The  calculation  of  the  collision  force  magnitude  will  be  investigated  further  to  improve 
haptic  rendering  at  locations  with  many  tool  object  intersection  as  they  appear  in  deep  clefts. 


Fig.  3:  Sub- voxel  rendered  anatomic  model 


Fig.  4:  Simulation  of  middle  ear  surgery 
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Abstract 

This  paper  outlines  the  major  motivating  factors 
concerning  a  novel  collaborative  project  between 
Edinburgh  College  of  Art  and  Edinburgh  Virtual 
Environment  Centre.  The  “ Tacitus ”  project  will 
investigate  the  use  of  multimodal  virtual  environments, 
specifically,  the  haptic  modality,  with  regards  to  the 
creative  processes  employed  by  designers  working  within 
the  field  of  applied  arts.  The  salient  areas  of  research  are 
described,  and  the  methods  by  which  information 
regarding  these  areas  will  be  obtained  are  considered. 
Initial  investigations  have  revealed  a  strong  need  to 
mimic  the  traditional  applied  artists’  workspaces,  with 
co-location  of  visual  and  haptic  cues  a  priority. 

1.  Introduction 

The  advent  of  new  virtual  media  and  haptic  computing 
means  that  technology  offers  not  only  new  tools  but  also 
the  prospect  of  entirely  new  ways  of  developing  work  in 
virtual  space.  Current  computer  interfaces  are  often 
alienating,  impoverished  worlds,  with  little  to  support  the 
intuitive  heuristic  working  practices  of  applied  artists. 
Applied  artists  are  designer  makers  and  artist  craftsmen, 
i.e.  people  who  work  creatively  with  their  hands,  using  a 
chosen  material  and  often  with  an  clement  of 
functionality.  Examples  of  applied  arts  disciplines  include 
ceramics,  glass,  jewellery,  furniture,  textiles,  metalwork, 
bookbinding,  and  calligraphy. 


Many  art  and  design  students  are  denied  the  opportunities 
presented  by  Information  Technology  (IT),  due  to  their 
fear  or  dislike  of  computers.  Perhaps  even  more 
importantly,  this  also  deprives  many  IT  based 
developments  of  their  potentially  valuable  input  [1]. 

Central  to  the  field  of  applied  arts  is  the  notion  of 
“intelligent  making”;  a  mix  of  formal  knowledge,  tacit 
knowledge,  physical  and  mental  skill,  contextual 
awareness,  innovation,  and  personal  creative  autonomy 
[2].  An  applied  artist’s  instinctive  grasp  of  constructing 
and  visualising  in  three  dimensions,  their  spatial  thinking 
and  sense  of  touch  are  integral  to  their  process  of 
creativity.  Makers  combine  all  their  sensory  modalities, 
such  as  sight,  hand  motions,  and  sound  in  order  to  explore 
and  bring  intended  qualities  to  the  object  they  are  making. 
Results  can  only  be  achieved  through  ongoing  dialogue 
between  the  maker,  materials  and  process  [3], 

In  October  2000,  with  the  above  indications,  Edinburgh 
College  of  Art  (ECA)  and  Edinburgh  Virtual 
Environment  Centre  (EdVEC)  began  a  three-year 
collaborative  research  project,  funded  by  the  Aits  and 
Humanities  Research  Board.  The  principal  aim  of  their 
user-centred  project  is  to  investigate  the  development  of 
three-dimensional  haptic  and  multi-sensory  computer 
applications  for  creative  processes  in  applied  arts  and 
design. 
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2.  Project  Overview 

The  main  aims  of  the  “Tacitus”  project  can  be 
summarised  as  follows: 

•  To  explore  the  potential  advantages  of  being  able 
to  work,  think  and  respond  to  physical  and  visual 
stimuli,  in  a  virtual,  fully  three-dimensional, 
non-gravity  context,  with  particular  reference  to 
the  education  of  designers  and  artists  and  the 
development  of  three-dimensional  work. 

•  To  discover  the  degrees  of  haptic  and  other 
multi-sensory  feedback  required  within  digital 
systems  to  assist  designers  and  artists  to  work 
more  intuitively. 

•  To  develop  viable  software  applications  and 
virtual  ‘hand  tools’  to  enhance  the  creative 
practice  of  applied  artists. 

Initial  studies  indicate  that  a  centra!  notion  to  the 
development  of  the  project  will  be  that  of  tacit  knowledge 
regarding  material  constraints.  In  the  case  of  the  creative 
process  of  applied  artists,  material  constraints  provide  an 
affordance  to  the  imagination,  rather  than  a  barrier.  An 
artist  must  understand  the  potential  of  materials,  tools  and 
techniques,  whilst  having  the  skill  to  control  the  actions 
required  to  achieve  creative  intent.  For  example  Figure  1 
illustrates  a  silversmith  forming  a  vessel  from  a  flat  sheet 
of  metal.  Audio,  tactile  and  proprioceptive  cues  are  all 
necessary  for  the  skilful  and  controlled  manipulation  of 
the  material. 


Figure  1.  Silver  vessel  in  the  process  of  being  formed 
from  flat  sheet. 


This  view  of  constraints  is  in  stark  contrast  to  the 
traditional  notion  of  virtual  reality  (VR),  which  allows  us 
to  interrupt  the  sensory  feedback  loop  with  the  perceived 


world,  and  therefore  facilitates  the  omission  of  material 
and  physical  constraints.  While  this  presents  many 
advantages  for  data  visualisation  and  manipulation,  to  the 
applied  artist,  it  removes  an  essential  catalyst  to  their 
creativity.  Hence,  research  will  initially  seek  to  identify 
material  properties,  tools  and  processes  that  are  salient  to 
the  creative  process  of  the  applied  artist. 

Glass  blowing  (Figure  2)  is  a  good  example  of  the 
complex  skills  and  tacit  knowledge  artists  possess  in 
regard  to  properties  and  constraints  of  a  material  and  tools 
used  to  manipulate  it.  Temperature,  viscosity  and 
adhesive  properties  must  all  be  judged  accurately  with 
split  second  timing  for  a  successful  outcome.  In 
transference  to  a  virtual  environment,  some  of  these 
variables  may  be  removed  or  controlled  with  benefit  to 
the  artistic  process.  However,  post  work  interviews 
determined  that  a  constraint  such  as  a  gravitational  field  is 
fundamental  to  the  tacit  knowledge  and  skills  of  these 
artists.  Hence  this  constraint  must  be  embodied  to  some 
degree  in  any  virtual  environment,  if  the  artist  is  to 
transfer  their  skills  and  tacit  knowledge  to  the  virtual 
workspace. 


Figure  2.  Skilful  co-operation  of  glass  artists  to 
produce  a  complex  artefact. 


3.  Methodology 

The  first  series  of  studies  will  seek  to  establish  a  data 
resource  on  fundamental  user  requirements,  in  order  to 
begin  developing  haptic  computing  for  applied  artists. 
These  are  in  progress  at  Edinburgh  College  of  Art,  first 
studying  students  and  staff  working  in  the  departments  of 
Design  and  Applied  Art,  followed  by  studies  of  selected 
professionals  working  in  a  variety  of  disciplines  and  using 
a  range  of  materials,  tools  and  skills. 


186 


Eurohaptics  2001  Birmingham  July  2001 


3.1  Key  Areas 

In  order  to  establish  a  meaningful  data  resource  for 
development  of  a  haptic  computing  system,  the  initial 
studies  will  be  focused  in  three  areas. 

A:  Conceptual  Approach 

The  pathways  into  the  initial  creative  process  of  design, 
e.g.  experimenting,  sketching,  modelling,  testing, 
searching,  playing.  Factors  in  the  design  process  that 
support  creativity. 

B:  Skills  /  Tools  /  Materials 

The  range  of,  and  preferred,  tools  and  materials 
employed,  and  the  creative  effects  that  can  be  achieved 
via  use  of  these.  The  material  tool  qualities  and 
properties,  and  the  skills  required  to  achieve  effects. 

C:  Computers 

Limitations  of  current  computer  graphics  and  3D  design 
applications,  for  example,  range  and  ini  ended  use  of 
available  design  software,  assessment  of  the  software’s 
capacity  to  accommodate  range  of  design  processes  and 
creativity,  and  a  “wish  list”  for  ideal  virtual  design  tools 
and  environment. 

3.2  Data  Acquisition 

3.2.1  Stimulated  Response.  Interview  during  the  design 
procedure  would  be  highly  disruptive  lo  the  artists’ 
thought  process,  and  difficult,  due  to  the  vast  amount  of 
information  that  must  be  conveyed  regarding  procedures, 
tools  and  materials.  Therefore,  photography  and  video  are 
currently  being  employed  to  obtain  a  record  of  artists’ 
working  methods  in  process,  as  opposed  to  staged 
demonstration.  Subsequent  review  of  the  tapes,  and  post¬ 
work  interviews  will  allow  detailed  investigation  of 
individual  artist’s  tacit  inferences. 

3.2.2  Questionnaires  /  Self-analysis.  Focussed  debate 
and  questionnaires  will  be  employed  to  encourage  ECA 
staff  and  students  to  articulate  the  reasons  for  preferred 
materials  and  tools,  and  virtual  environments  that  would 
help  to  enhance  the  creative  process.  Two  of  the 
researchers  on  the  project  are  applied  artists  of 
international  status.  Their  self-observation  during 
initiation  and  development  of  their  own  work  will  provide 
a  valuable  resource. 

3.2.3  Further  Studies.  Further  stages  of  the  research 
project  will  investigate  tacit  knowledge,  hnptic  and  other 


senses.  Data  from  these  studies  will  be  used  in  an  iterative 
process  to  evolve  digital  systems  to  assist  designers  and 
artists  to  work  more  intuitively  and  support  development 
of  virtual  hand  tools  and  viable  software  application  to 
enhance  creative  practice. 

4.  Initial  Findings 

4.1  Haptic  Device  Implementation 

The  specific  aims  of  the  project  dictate  the  choice  of 
haptic  device  to  be  employed  to  a  high  degree.  The  artists 
and  designers  who  would  use  the  equipment  had 
subjective  requirements  on  the  method  of  operation, 
which  also  constrained  choice.  This  required  that  the 
virtual  object  being  manipulated,  and  the  tools  being  used 
to  do  so,  should  occupy  a  position  in  physical  space 
similar  to  that  when  working  in  real  life  and  that  the  mode 
of  working  should  feel  the  same. 

The  equipment  finally  chosen  also  had  to  have  reached  a 
commercial  stage  of  development  with  adequate  support 
and  have  a  reasonable  development  environment. 

The  Reachin  (www.reachin.se)  system,  based  on  the 
PHANToM  (www.sensable.com)  haptic  device  was 
chosen  for  the  project.  The  main  reasons  for  this  choice 
were  the  relative  product  maturity  of  the  PHANToM,  the 
co-location  design  of  the  real  probe  with  its  virtual 
depiction,  and  the  high  level  Reachin  API  development 
environment.  Figure  3  illustrates  the  high  degree  of 
similarity  present  between  a  typical  jeweller’s  workbench, 
and  the  Reachin  system. 


Figure  3.  Comparison  between  traditional  jeweller’s 
working  position  and  co-Iocated  haptic/visual  display. 


4.2  Questionnaire  Results 

A  general  questionnaire  completed  at  project 
presentations  by  ECA  students,  and  by  professional 
applied  artists  revealed  the  following  points  concerning 
computer  and  tool  use  within  the  field  of  applied  arts: 

•  A  wide  variety  of  preferred  hand  tools,  therefore  the 
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need  to  categorise  these  tools  so  that  generic 
properties  across  the  disciplines  can  be  extracted  and 
analysed. 

•  Of  ECA  students  across  all  disciplines  in  applied  arts, 
46%  use  computers,  software  named  included 
AutoCAD,  CorelDraw  and  Photoshop 

•  All  students  in  furniture,  but  no  students  in  jewellery 
and  metalwork,  used  computers  for  designing. 

•  Of  professional  applied  artists  invited  and  attending  a 
presentation  about  the  project,  84%  use  computer 
with  23%  specifically  for  design.  This  high 
percentage  could  be  reflected  in  the  interest  of  those 
attending  in  computer  applications. 

These  preliminary  results  show  a  relatively  low  use  of 
computers  within  the  applied  artist  community,  while 
there  was  a  wide  variety  of  hand  tools  employed.  It  was 
also  observed  that  single  tools  are  often  used  for  a  number 
of  different  purposes  (e.g.  a  welding  torch  can  be  used  to 
cut,  bend  or  join  material),  and  are  therefore  very  flexible 
in  the  hands  of  a  maker.  This  suggests  that  the  inherent 
rigidity  of  current  software  tools  is  a  barrier  to  the 
creative  process. 

5.  Conclusion 

This  paper  has  outlined  the  primary  research  goals  of  the 
Tacitus  project,  which  seeks  to  develop  a  multimodal 
virtual  environment  capable  of  meeting  the  needs  of 
applied  artists,  craftsmen  and  designers.  In  particular,  the 
project  will  focus  on  the  use  of  tacit  knowledge  by  artists 
and  designers,  and  the  role  of  material  constraints  and 
properties  in  the  creative  process. 

Our  goal  is  not  to  imitate  the  working  practices  and 
environment  of  the  craftsman,  but  to  create  a  generic 
virtual  environment  that  can  be  applied  to  a  variety  of  3D 
creative  disciplines,  in  which  the  applied  artist  teels 
comfortable  and  uninhibited  by  the  novel  synthetic 
environment  and  yet  can  bring  their  experience  and 
knowledge  to  extend  their  levels  of  creativily  more  fluidly 
using  a  new  digital  medium. 
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Abstract 

In  this  paper  we  present  a  novel  approach  to  communi¬ 
cation  using  the  human’s  tactile  sense.  We  propose  an 
intelligent  tactile  information  system  that  has  the  ca¬ 
pability  to  generate  a  language  suitable  for  tactile  stim¬ 
ulation.  This  unknown  language  has  to  be  evolved  by 
an  adaptive  encoder  in  interaction  with  the  user.  The 
system  is  composed  of  four  functional  units:  input  de¬ 
vice,  learning  encoder,  tactile  interface  and  mans  per¬ 
ception. 

1  Introduction 

Human  beings  communicate  with  their  natural  and 
technical  environment  by  using  different  channels  for 
information  exchange,  e.g.  seeing,  hearing,  smelling, 
touching,  speaking,  gesturing  etc.  Most  part  of  human 
perception  is  processed  through  the  visual  and  audi¬ 
tory  channel,  while  the  tactile  channel  remains  rather 
unused.  Although  seldom  used  for  technical  supported 
communication  tactile  perception  is  a  powerful  infor¬ 
mation  channel  e.g.  demonstrated  by  the  Tadoina 
method  [8]  or  by  the  users  of  braille  displays. 

There  are  versatile  applications  using  the  tactile 
sense  [2]  [4]  [11].  Handicapped  people  can  use  the  tac¬ 
tile  channel  as  substitute  for  the  lost  channel  of  per¬ 
ception  [3].  As  an  additional  possibility  private  com¬ 
munication  is  possible  because  it  is  non-audible  and 
non-visible  and  therefore  not  easily  monitored  by  un¬ 
wanted  listeners.  Furthermore  the  tactile  sense  can  be 
used  as  a  spatial  orientation  guide. 

It  is  reasonable  to  subdivide  the  flow  of  information 
in  a  tactile  information  system  from  the  real  world  to 
human  perception  into  four  units:  an  input  device,  an 
encoder,  a  tactile  interface  arid  human  perception. 

The  input  device  receives  the  signals  and  transforms 


them  into  a  convenient  electronic  representation.  The 
encoder  processes  the  information  in  a  suitable  way  so 
that  the  tactile  interface,  the  connection  to  the  human 
being,  can  be  optimally  controlled.  The  final  unit  of  a 
tactile  information  system  is  the  individual  itself  with 
its  cortical  information  processing  that  finally  enables 
perception. 

The  complete  information  process  from  the  input  de¬ 
vice  to  human  perception  is  extremly  complex  and  fur¬ 
thermore  unknown.  At  the  moment  it  is  impossible  to 
model  this  information  process  in  a  sufficient  way.  Thus 
we  recommend  that  the  adequate  information  process¬ 
ing  is  attained  by  an  adaptive  system  in  close  interac¬ 
tion  with  a  learning  scheme. 

We  propose  tactile  systems  should  consist  of  an  input 
device,  an  adaptive  encoder  [5],  an  actuator  and  human 
perception.  The  adaptive  encoder  is  trained  in  special 
learning  sessions  [1]  to  obtain  the  required  capabilities. 


2  Architecture 

The  adaptive  tactile  system  consists  of  four  units  as 
depicted  in  figure  1:  input  device,  adaptive  encoder, 
tactile  interface  and  perception  by  the  user. 


*  The  INPUT  DEVICE  is  designed  to  pick  up  the 
signals  from  the  input  space.  There  are  differ¬ 
ent  input  spaces  possible  e.g.  acoustical,  visual 
spaces  or  data  sources  (computer).  It  is  rea¬ 
sonable  to  preprocess  the  input  data  to  achieve 
that  the  time  for  the  adaption  of  the  encoder 
becomes  short.  For  example  acoustical  input 
could  be  filtered  and  classified  into  patterns  of 
phonemes  [9]  [10]  to  reduce  dimension  of  the  data 
without  destroying  the  relevant  information. 
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Real  world  Input  device 


Encoder  Tactile  Interface  User’s 


Figure  1:  Scheme  of  tactile  information  system,  consisting  of  Input  Device,  adaptive  Encoder,  Actuator,  User’s  Perception 


•  The  ENCODER  obtains  the  preprocessed  data 
from  the  input  device.  It  processes  the  data  arid 
translates  the  information  into  a  language  ade¬ 
quate  for  tactile  human  perception.  At  the  mo¬ 
ment  this  language  is  unknown  and  thence  the 
translation  is  impossible  to  calculate.  A  neural 
network  with  its  approximation  capabilities  offers 
a  feasible  approach  to  solve  this  problem.  Thus 
the  encoder  becomes  versatile  and  adaptive  with 
respect  to  the  type  of  input  signal  and  the  indi¬ 
vidual  user’s  perception. 

•  The  TACTILE  INTERFACE  is  the  connector 
from  the  encoder  to  the  user.  The  stimulators 
of  the  interface  can  be  of  different  types  such  as 
vibratotactile  or  electrotactile  [6]  [7].  In  the  skin 
exist  diverse  types  of  mechanoreceptors.  To  ob¬ 
tain  a  higher  bandwidth  of  information  it  is  ad¬ 
vantageous  to  be  able  to  excite  the  receptors  sep¬ 
arately.  Additionally  the  consumption  of  power 
should  be  low  because  the  tactile  information  sys¬ 
tem  needs  to  be  portable. 

»  The  USER  is  the  unit  of  the  system  where  the 
perception  is  finally  generated.  Tactile  percep¬ 
tion  will  be  different  for  different  users  so  that  the 
individual  adaptation  of  the  encoder  is  a  great  ad¬ 
vantage.  For  this  adaptation  feedback  of  the  user 
is  required.  The  presentation  of  the  feedback  de¬ 
pends  on  the  implementation  of  the  learning  al¬ 
gorithm.  The  user  too  has  to  adapt  himself  in  a 
way  that  he  learns  to  interpret  the  presented  tac¬ 
tile  stimulation.  The  aim  is  to  achieve  a  sensor y 
substitution.  Then  the  former  tactile  perception 
is  replaced  by  a  new  type  of  perception  resulting 
for  example  in  an  ability  of  hearing  by  means  of 
the  tactile  sense. 


3  Handling  the  system 

We  propose  to  install  the  tactile  system  in  three  phases: 

•  In  the  first  phase  the  user  has  to  be  sensitized  to 
this  specific  type  of  stimulation. 

•  During  the  second  phase  the  system  is  adapted. 
For  this  purpose  a  finite  set  of  the  encoder’s  input 
signals,  representative  for  the  input  space  of  the 
tactile  system,  is  generated.  To  allow  the  user  to 
distinguish  and  learn  the  presented  patterns  as 
good  as  possible,  the  parameters  of  the  encoder 
are  adapted  by  the  learning  algorithm  having  re¬ 
gard  to  the  user’s  appraisal  of  the  stimulation 
(feedback).  As  there  are  two  learning  parts  in 
the  tactile  system  (adaptive  encoder  and  user) ,  a 
learning  scheme  has  to  be  carefully  developed  so 
that  the  two  parts  interfere  in  a  constructive  way. 
Although  presenting  only  a  finite  set  of  input 
patterns,  the  encoder  should  afterwards  be  able 
to  map  unknown  but  similar  patterns  to  a  tac¬ 
tile  stimulation  that  can  be  recognized  and  inter¬ 
preted  by  the  user.  This  generalization  is  a  well- 
known  feature  of  the  neural  network  implemented 
in  the  encoder. 

•  Finally  the  system  is  adjusted  and  ready  for  day- 
to-day  use.  It  may  be  necessary  to  readjust  the 
system  from  time  to  time. 

4  Conclusion 

The  introduced  tactile  information  system  offers  an  ef¬ 
ficient  way  of  communication.  It  establishes  an  addi¬ 
tional  channel  of  information  exchange.  With  the  use 
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of  an  adaptive  encoder  the  tactile  information  system 
evolves  a  language  suitable  for  tactile  stimulation.  By 
the  aid  of  sensory  substitution  the  information  is  re¬ 
ceived  by  the  subconscious  mind  and  does  not  divert 
the  user  from  other  tasks.  With  different  types  of  input 
spaces  the  tactile  information  system  offers  a  variety  of 
novel  fields  it  can  be  applied  to. 
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'Words  and  images  behave  in  different  ways.  A  picture  of  a  miniskirt  performs  a  different  Junction  than  the  naming  of  it. 
Barthes  suggest  that  words  make  the  image  “intelligible:  [that]  it  is  not  the  object  but  the  name  that  creates  desire  (Barthes 
1990) ...  Orality  implies  community.  Not  only  do  we  dress  up;  we  talk  about  it’ 111 


Abstract.  The  above  suggests  ‘oral  texture’  as  a  method  of  communication  or  'ideal  and  emotive  novelisation’ ,  a  utopian  standard  of 
image  and  self-transformation,  set  by  the  fashion,  textiles  and  lifestyle  perception  makers.  It  is  a  system  that  has  been  used  to  sell  or 
seduce  magazines  (imagery  and  supporting  text)  to  the  reader  (wearer)  since  the  late  19°'  Century  and  blurred  into  the  realms  of  retail 
sales  and  marketing,  especially  on-line.  Society  understands  and  indulges  in  this  method  of  aesthetic  communication,  and  for  women, 
it  is  meant  to  represent  'women 's  mass  culture’.  [Naomi  Wolf]1" 

This  paper  describes  two  fabric-touch  experiments  where  the  term  'oral  texture’  applies  in  the  collection  of  adjectives  and  metaphors 
(fabric  perceptions).  The  purpose  of  this  study  was  to  qualitatively  record  subjective  responses  to  familiar  fabrics,  using  a  Repertory 
Grid  (factor  analysis)  model  Bl,  to  preliminary  establish  subjective  and  objective  responses  to  fabrics,  and  through  analysis,  the  factors 
underlying  fabric  perception  and  discrimination  between  fabric  ‘types’. 

10  fabrics  were  selected.  Each  subject  evaluated  fabrics  using  their  index  finger  in  a  forward  and  backward  motion,  with  visual 
observation.  The  experiment  was  carried  out  in  three  phases.  The  first  experiment  was  used  as  a  Pilot  Study.  Data  was  assembled 
from  1  experienced  textile/fashion  professional/academic.  The  second  round  of  data  was  assembled  from  20  female  subjects  who  are 
currently  1“  year  fashion  and  textile  design  students.  Through  the  experience  of  the  pilot  experiment,  it  was  found  that  these 
qualitative  responses  were  split  into  two  categories,  1.  Surface  Texture  and,  2.  Emotional/Cognitive/Mood  associations.  After 
analysis,  principal  factors  for  each  category  evolved.  These  factors  were  then  labelled.  Results  are  summarised  and  discussed. 


Using  the  “experiment”  experience,  the  students  were  than  asked  to  select  their  own  fabric  of 
choice  and  evaluate  it  in  a  similar  manner  (this  was  not  limited  to  evaluation  by  index  finger,  as 
the  experimenter  was  not  present.  After  discussion  it  was  found  that  the  method  of  multiple 
finger  pinch,  as  illustrated  in  Figure  1  was  primarily  used,  it  being  the  most  natural  method  of 
touch  next  to  ‘touch-stroke’  (see  Figure  9) |3i.  Qualitative  data  was  collected,  as  before.  They 
were  then  asked  to  visually  respond  in  a  “virtual”  manner  by  creating  a  “mood  board”  using 
Adobe  PhotoShop  (a  combination  of  selective  imagery  related  to  texture,  colour  and  style, 
evoking  emotion  and  memory).  This  work  is  also  discussed  and  some  examples  are  shown  later 
on  in  the  paper. 

1.  Introduction 


1.1  Background 

When  a  person  runs  their  finger  across  the  surface  of  a  fabric,  or  when  we  travel  out  on  a  shopping  trip  for  clothing 
where  we  engage  in  a  selection  process  that  involves  touching  and  trying  on  clothing,  a  complex  multi-sensory, 
emotional  and  cognitive  experience  takes  place.  A  memory  is  stirred,  an  emotion,  feeling  and  association  is  evoked 
and  a  decision  is  made,  an  impression  becomes  embossed  in  the  mind.  When  we  shop  for  clothes  for  example,  this 
eruption  of  activity  extends  itself  into  a  manifestation  of  building  and  development  of  the  ‘self.  Decisions  and 
motivations  are  based  on  anticipated  reality  of  preference,  personality,  emotion  and  moods,  for  audience  or  non¬ 
audience  participation. 

In  terms  of  fabric/texture,  clothing/fashion  perception  and  virtual  technologies,  little  has  been  established  that 
sufficiently  deals  with  this  complex  relationship.  Within  the  realms  of  technology,  it  is  easy  to  be  misguided.  Human 
factors  can  become  disguised  or  blurred  into  generalisation  or  unimportance  when  ideally,  a  complementary  strategy;  a 
human-centred  system  is  necessary  that  combines  novel,  therapeutic  and  logical  variables.  A  system  that  involves 
textiles  and  fashion,  will  rely  heavily  on  emotion,  motivation,  creativity,  abstract  and  illogical  input  (human).  “People 
excel  at  qualitative  considerations,  machines  at  quantitative  ones.  As  a  result,  for  people,  decisions  are  flexible 
because  they  follow  qualitative  as  well  as  quantitative  assessment,  modified  by  special  circumstances  and  context.  For 


Figure  1.  Finger  Pinch 
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the  machine,  decisions  are  consistent,  based  upon  quantitative  evaluation  of  numerically  specified,  context-free 
variables.  Which  is  to  be  preferred?  Neither:  we  need  both".  141  If  the  machine  can  learn  as  we  do,  see  as  we  do,  feel 
as  we  do  then  the  bridge  between  people  vs.  machines  can  be  broken  and  their  role  as  ‘assistant’  be  reconciled  with. 

The  emphasis  of  this  research  is  in  the  early  development  of  a  valuable  tool  that  eventually  retailers,  designers  and 
consumers  should  all  potentially  benefit  from. 

1.2  Methods  and  Analysis 

The  repertory  grid  theory  has  been  used  successfully  to  study  consumer  behaviour.  “Kelly  (1955)  proposed  that  people 
act  on  the  basis  of  specific  hypotheses,  or  expectations,  concerning  the  functioning  of  their  environment,  i.e.  people  are 
assumed  to  be  ‘scientists’,  developing  hypotheses  concerning  the  best  course  of  action  to  take  in  a  given  situation”  1  . 
This  has  been  labelled  a  “mental  model”  t21,  processes  through  which  people  evaluate  their  own  surroundings  and 
products  and  who  possess  different  levels  of  experience  and  expertise.  The  theory  is  based  on  the  assumption  that 
people  focus  on  positive  and  negative  or  reward  and  punishment  aspects  for  any  given  experience.  In  terms  of  fabric 
perception  we  will  say  the  ‘opposites’,  however  in  terms  of  subjective  emotion  we  may  also  say  positive  or  negative. 
The  theory  was  labelled  “personal  construct  theory”  '2|.  A  method  of  ‘triads’  is  used  to  collect  data.  On  this  occasion 
for  example,  the  experimenter  presented  1 0  different  fabrics  (product)  where  the  individual  had  to  select  three  fabrics  at 
a  time,  decide  which  two  were  the  most  similar  and  describe  why  they  differed  from  the  third.  This  was  performed  in 
rotation  until  all  possible  combinations  had  been  covered  (10).  For  example  if  you  have  a  fine  silk,  a  velvet  and  a 
leather  it  could  be  said  that  the  silk  and  the  velvet  are  the  pair,  both  being  incredibly  smooth,  soft,  silky,  reflective, 
glossy  and  somewhat  oily  to  touch.  The  leather,  even  though  it  is  smooth  is  different,  it  feels  moist,  matty  and  spongy 
to  the  touch.  In  terms  of  emotion/cognitive/other,  the  velvet  and  the  silk  could  be  seen  as  feminine,  gentle,  sensuous, 
thoughts  of  beauty,  weddings,  and  party  dresses,  majestic  happiness,  decadence,  gothic  styles  and  old  dusty  houses 
spring  to  mind,  whereas  the  leather  could  be  deemed  as  masculine,  sexual,  feelings  of  power,  fetishism,  being  tough, 
strong,  possessing  animal  instincts,  cowboys,  the  night,  it  can  be  deemed  somewhat  threatening  -  leather  clothing, 
bohemia  and  wild  rock  chic’s  attitudes,  and  a  sense  fearlessness  when  worn,  see  Figure  2,  below  '5|. 


Figure  2. 

‘There’s  no  doubting  that  leather  represents  masculinity  and  that's  what  our  girls  want 
(regardless  of  their  sex)’  (Robert  Stoller,  Observing  the  Erotic  Imagination,  1985  [6) 

‘Leather  still  emerges  with  a  tough,  give-  ‘em-hell  attitude,  which  should  satisfy  the  Thelma 
and  Louise  in  every  woman... '  (Anna  Wintour,  Vogue,  September  1991)  1,1 

Principal  components  analysis,  a  multivariate  analysis  technique,  (originated  by  Pearson, 
1901  and  developed  by  Hotelling  -  1933)  171  was  used  here.  Given  a  set  of  observed 
uncorrelated  (unrelated)  variables  the  data  was  reduced  down  to  the  underlying  dimensions 
based  on  total  variation,  providing  fewer  variables  or  correlated  ‘principal  components’  to 
examine  and  be  used  as  an  objective  data  source. 

1.3  Fabric  and  Clothing/Fashion  Perceptions 

One  of  Lederman’s  reasons  for  why  our  knowledge  of  touch  has  not  significantly  developed  as  rapidly  as  our  visual 
sense  is,  “there  is  a  general  reluctance  in  our  society  to  discuss  touch-related  matters”. 181  Vision  seemingly  dominates 
our  attention  whereas  touch  can  therefore  be  seen  as  a  private  and  complex  emotional  and  cognitive  experience, 
especially  when  anticipating,  seeing  and  feeling  the  skin  /  body  in  touch  with  fabric  via  clothing  (image  creation),  a 
‘second  skin’.  151  Encased  within  our  clothing,  unconsciously  or  consciously  our  individual  arrays  of  texture,  colour, 
design  and  style  choices  act  as  a  metaphor  for  the  self  affecting  our  behaviour,  emotion  and  moods,  levels  of 
motivation,  self  esteem/confidcnce  representing  itself  as  a  visual  and  tactile  mirror  of  the  self,  of  our  society  and 
culture. 1,01 


Through  history,  for  example  in  the  early  19th  Century  England,  the  textures  favoured  at  that  time,  '  indicated,  for  men, 
both  natural  (Greek)  man  and  virtue;  for  women,  childish  frailty’. 


‘Wool:  National  pride,  simplicity,  modern  revision  of  classicism,  cult  of  nature;  matte  texture  is  tantamount  to  smooth 
skin  and  natural  virtue  (Sir  John  Reynold's  objection  to  Michelangelo's  "play  of  light  over  rich  texture'  that  inspired 
sensuality);  mallability  allows  it  to  be  molded  to  a  man's  curves;  moral  strength. 

Broadcloth  (on  women):  Androgyny 
Wood:  Natural  man,  modern  craftsmanship 
Leather:  National  pride,  shiny  foot-fetish  fodder,  naturalism 
Cotton:  Colonial  acquisition 

Silk:  Exotic  /  innocent  (depending  on  how  it  is  woven)  for  women;  for  men,  outdated,  Rocco,  wrinkling,  light-reflecting 
focus  on  artifice  rather  than  on  nature 
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Muslim  (white,  for  women):  Waiftsh  innocence;  ditto  bouncing  hairm 

However,  ‘In  her  study  on  fetish  clothing,  Valerie  Steele  argues  that  since  the  nineteenth  century  there  has  been  a  slow 
but  definite  turn  to  the  so  called  masculine  "hard"  materials,  especially  leather  and  rubber.  With  the  beginning  of  the 
modern  age,  she  concludes,  the  need  for  a  certain  materiality  constantly  grows,  and  apparently  it  has  to  fulfil  a  specific 
desire  for  a  contact,  which  can  only  be  expressed  in  a  tactile  and  haptic  manner’. 161 

This  desire  has  become  somewhat  unisex,  especially  in  terms  of  the  growth  in  wearing  casual  inspired  clothing  and  in 
the  development  of  interactive,  smart  fabrics  and  wearable  computers  that  are  promised  to  enhance  lifestyles  for 
convenience  sake.  The  term  ‘cyborg’  is  fast  becoming  a  reality. 

However,  the  other  perspective,  ‘fashion’  based  a  constant  change  of  trends,  will  always  remain,  ‘To  transform  yourself 
into  a  Romany  princess  is  divine  -  if  it  turns  you  up  and  puts  you  at  your  ease.  But  don’t  get  lost  in  someone  else’s 
dream:  there  is  enough  adventure  in  fashion  to  find  your  own’  (1970)  Diana  Vreeland,  Editor  in  Chief  of  Vogue,  1963- 
1970 111 

1.4  Aims  &  Objectives 

The  aim  is  to  develop  a  multi-modal  virtual  reality  model  that  incorporates  a  tactile  (haptic)  dimension,  and  to  satisfy 
and  soothe  the  physical  perceptions,  responses  and  emotions  related  to  motivations  for  wearing  and  selecting  clothes 
(essentially  aspects  of  ‘the  self’).  Existing  and  developing  technology  has  the  potential  to  offer  solutions  to  this 
problem. 

There  are  variables  associated  with  this  hypothesis.  These  can  be  categorised  as  follows:  fabric/texture  perception  and 
motivation,  style/design  perception  and  motivation  both  tied  to  self/self-image  (personality  and  emotional  response). 1121 
This  study  provides  some  insight  into  what  motivates  an  individual  in  terms  of  fabric  and  texture  and  some  of  the  social 
implications  involved. 

This  research  could  be  used  to  refine  texture  simulation  in  haptic  technology  and  for  an  objective  or  subjective  non¬ 
verbal  communication  system  within  a  virtual  environment 

2.  Methods 

2.1  Pilot  Experiment 


Stimuli.  10  different  fabrics  (familiar),  listed  as  follows,  mounted  flat  on  MDF  wood  measuring  3"  x  3",  taped  down 
at  two  sides: 


1 .  Fleece  ( 1 00%  Polyester) 

2.  Lycra  (83%  Polyamide,  17%  Elastane) 

3.  Sheepskin 

4.  Silk  (100%  Silk) 

5.  Corduroy  ( 100%  Cotton ) 
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6.  Leather 

7.  Velvet  (1007c  Viscose) 

8.  Irish  Linen  (100%  Cotton) 

9.  Denim  ( 1007c  cotton) 

10.  Lace  (657c  Polyamide,  357c  Cotton 

Subjects.  1  textile/fashion  professional  (academic/research). 

Procedure.  The  subject  was  tested  individually  in  three  phases.  The  subject  sat  at  a  table,  and 
using  her  right  arm  extended  forward  with  arm  resting  on  table.  For  evaluation  of  the  fabrics 
presented,  with  visual  observation,  the  subject  was  instructed  to  stroke  using  her  index  finger 
making  a  fist  with  the  other  fingers  (Figure  4.).  As  outlined  briefly  above,  using  the  Repetory 
Grid  model,  the  10  fabrics  were  presented  to  the  subject.  The  subject  was  then  asked  to  select 
three  fabrics  at  random  (triads)  and  to  continue  to  do  this  until  at  least  10  different  combinations 
had  been  covered.  The  subject  had  to  decide  which  two  fabrics  out  of  the  three  were  most  alike 
and  provide  adjectives  (constructs)  as  to  how  they  differed  from  the  third  fabric.  Seeing  the  same 
fabric  in  different  combinations  (contrasts)  allows  one  to  notice  other  perceptions  in  terms  of 
texture  and  emotion/moods  that  may  not  have  been  considered  on  initial  evaluation  of  triad 
combinations.  The  experiment  took  place  in  a  university  studio  environment.  Classic  FM  was 
playing  softly  in  the  background  to  open  up  the  emotions  and  relax  the  subject.  This  was  turned 
off  at  a  later  stage  without  any  comment.  Subject  tended  to  close  eyes  to  concentrate  on  ‘touching’  the  fabrics  and 
experienced  sleepy  tendencies  (winter  time).  As  expected,  colour  tended  to  add  to  the  emotional  impact  even  though 
the  focus  was  on  ‘touch’. 

As  outlined  earlier,  the  constructs  automatically  split  into  two  categories,  which  described  1.  Fabric  Surface,  and  2. 
Emotional/Cognitive/Mood  associations.  This  phase  took  approximately  1  hour.  After  tabling  data,  in  the  second 
phase,  the  subject  was  presented  with  the  10  fabrics  again  and  asked  to  indicate,  through  the  same  touch  and  visual 
method,  agreement  with  an  individual  fabric  and  a  construct  entering  a  “1”,  if  the  construct  was  not  present,  they 
indicated  this  with  a  “0”.  This  took  approximately  35  minutes  depending  on  how  many  adjectives  were  originally 
given  by  the  individual. 

A  Principal  Component  Analysis  using  SPSS  was  run  and  the  Construct  Groupings  (Factors)  evolved.  On  a  separate 
day,  for  Phase  3  of  the  experiment,  the  subject  was  then  asked  to  label  each  factor. 

The  purpose  of  the  pilot  experiment  was  to  learn  how  subjects  would  describe  stimuli  when  given  no  other  guidelines 
except  a  request  for  adjectives,  consequently  providing  a  model  to  follow  for  the  main  experiment. 

Results.  Results  are  summarised  in  Table  1  below. 


Table  1.  Pilot  Study  Factors 


I  Fabric  Surface 

Factor 

Construct  Groups 

Factor  Label 

1 

Resistant,  soft,  very  soft,  warm,  quite  warm,  animal,  deep  pile,  hairy 

Downy 

2 

Changeable  surface,  high  friction,  uneven  surface 

Irregular 

3 

Quite  silky 

Velvety 

4 

Stretchy,  non-rhythmic 

Flexible 

5 

Spongy,  grainy 

Yielding 

6 

Bubbly,  non-skin  like 

Artificial 

7 

Silky,  non-gritty 

Smooth 

Emotional/Cognitive/Moods 

1 

Stimulating,  very  nice,  caring,  secure,  luxurious,  calming,  content,  sleepy 

Satisfied 

2 

Teddy  bears,  warm 

Loving 

3 

Comforting,  babies,  happy,  non-painful,  active 

Compassionate 

4 

Relaxed 

Tranquil 

5 

Sensual,  disorganisation 

Blurred 

Figure  4.  Index- 
Finger-Stroke 


For  “Emotional/Cognitivc/Mood”  relationships,  there  remained  3  constructs  -  Cosy,  Efficient  and  Childish.  However 
it  is  acceptable  that  Cosy  and  Childish  is  covered  in  Factor  2  and  3  and  Efficient  in  Factor  1.  The  above  data  provides 
acceptable  subjective  results. 

The  main  experiment,  below,  provides  objective  rather  than  subjective  results  through  group  analysis. 
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2.2  Group  Study 


Stimuli.  10  different  fabrics  (familiar),  listed  as  follows,  mounted  flat  on  MDF  wood  measuring  2"  x  3",  taped  down  at 
two  sides: 


Figure  5.  Fabrics  for  Group  Study 


1.  Denim  (100%  Cotton) 

2.  Fleece  ( 100%  Polyester) 

3.  Corduroy  ( 100%  Cotton) 

4.  Lycra  (83%  Polyamide,  17%  Elastane) 

5.  Velvet  (100%  Viscose/Silk) 

6.  Satin  Silk  (100%  Silk) 

1.  Irish  Linen  (100%  Cotton ) 

8.  Tweed  (700%  Wool ) 

9.  Lace  (65%  Polyamide,  35%  Cotton) 

10.  Leather 

Subjects.  20  female  1"  Year  Fashion  &  Textile  students  aged  19-23,  including  two  mature  students  aged  between  30- 
40. 

Procedure.  The  procedure  was  set  out  as  in  the  Pilot  experiment  above;  the  environment  was  a  university  studio 
setting.  For  Phase  3  (labelling  of  factors),  4  of  the  students  and  the  experimenter  (5)  sat  down  in  a  brainstorming 
session  to  provide  labels  for  the  ‘Fabric  Surface’  factors  that  evolved.  The  experimenter  gave  labels  for  the 
Emotion/Cognitive/Mood  responses. 

Results.  Five  main  factors  were  extracted,  accounting  for  87.2%  of  the  variance.  Items  were  allocated  to  factors  if 
their  correlation  exceeded  0.75  for  Factors  1  and  2  and  0.5  for  3-5. 

Results  are  summarised  in  Table’s  2-6,  below. 


Table  2.  Group  Study  Factors 


Fabric  Sur 

face 

Factor 

Construct  Groups 

Factor  Label 

1 

(34.5%) 

-rCoarse,  +friction,  -(-sandpaper,  +thick 

Rough 

-Silky,  -very  soft, 

Downy 

+/- 

Animal 

2 

(26.7%) 

-(-Stiff,  +very  smooth,  +waterproof,  +furry 

Coated  Seabed 

-Breathable,  -clean 

Fresh 

+/- 

Protective 

3 

(13.2%) 

+Natural,  -t-tingly,  +spongy,  +textured 

Mossy 

4 

+Gaps,  +rough,  +not  smooth 

Coarse 
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(6.8%) 

5 

(6.0%) 

-t-Closc  weave,  +slippery 

Silky 

Fmotional/Cognitive/Moods 

Factor 

Construct  Groups 

Factor  Label 

1 

(19.52%) 

-t-lly  a  fire,  +country  walks,  +feminine,  +hats,  +warm,  +winter 

Sedate  Warmth 

2 

(12.92%) 

+  19S0’s,  +  modern 

Contemporary 

-Ancient,  -nostalgic 

Eternal  Melancholy 

+/- 

Faded  Familiarity 

3 

(10.48%) 

+Comfort,  -bikers,  -masculine,  -trousers 

Male 

4 

(8.24%) 

•(-Cycling  shorts,  +dancc  clothing,  +leotards,  +swimming 

Energised 

5 

(6.67%) 

+  Rich,  -summer  shirts 

Opulent  Poise 

[+  =  positive  correlation,  -  =  negative  correlation] 


For  “Fabric  Surface”,  Factors  1  and  2  cover  the  most  variance.  Fabrics  were  therefore  plotted  as  Factor  1  against 
Factor  2  as  follows.  This  offers  a  somewhat  contemporary  and  stringent  approach  in  correlating  Fabric  Surface 
responses,  but  an  unsettling  approach  in  terms  of  emotion/cognitive/mood  responses: 


Table  3.  Factor’s  1  &  2  Allocation 


Fabric  Surface 

Factor  1:  Animal 

Factor  2:  Protective 

Denim 

Fleece 

Corduroy 

Lycra 

Velvet 

Irish  Linen 

Silk 

Leather* 

Tweed 

Lace 

Factor  2:  Faded  Familiarity 

Fleece 

Denim 

Corduroy 

Lycra 

Silk 

Velvet 

Tweed 

Irish  Linen 

Leather 

Lace 

*  Leather  could  also  come  under  Factor  1 ,  but  ultimately  belongs  in  Factor  2. 


Using  all  5  factors,  results  are  summarised  as  follows: 


Table  4.  Five-Factor  Allocation 


Fabric  Type 

Corduroy,  Leather 

Protective 

Fleece 

Mossy 

Tweed,  Denim 

Coarse 

Lace 

Silky 

Lycra,  Velvet,  Satin  Silk,  Irish  Linen 

Emotion/Cognitive/Mood  Factors 

Fabric  Type 

Sedate  Warmth 

Fleece,  Tweed 

Faded  Familiarity 

Satin  Silk,  Lace, 

Male 

Corduroy,  Leather 

Energised 

Lycra,  Denim 

Opulent  Poise 

Velvet,  Irish  Linen 

Finally,  if  we  merge  both  sets  of  factors  together,  we  have  the  following  summary: 


Table  5.  Combined  Factors 


Emotioii/Cogniti  ve/Mood 

Fabric  Surface 

Sedate  Warmth 

Protective 

Faded  Familiarity 

Coarse 

Male 

Animal 

Energised 

Mossy 
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Opulent  Poise 

Silky 

Fabric 

Combined  Factors 

Denim 

Energised  -  Mossy 

Lycra 

Energised  -  Mossy 

Fleece 

Sedate  Warmth  -  Protective 

Corduroy 

Sedate  Wannth  -  Protective 

Leather 

Male  -  Animal 

Velvet 

Opulent  Poise  -  Silky 

Satin  Silk 

Opulent  Poise  -  Silky 

Irish  Linen 

Opulent  Poise  -  Silky 

Tweed 

Faded  Familiarity  -  Coarse 

Lace 

Faded  Familiarity  -  Coarse 

This  study  evaluated  fabric  responses  and  suggests  that  the  characteristic  “feel”  of  a  given  fabric  depends  on  a 
particular  combination  of  variables  involving  both  a  fabric  surface  and  its  emotion/cognitive/mood  associations.  If  we 
label  them  separately  for  example  categorising  a  fabric  under  a  fabric  surface  factor  only,  the  fabric  type  can  differ  in 
the  emotional  response.  In  consideration  of  the  variables  that  make  up  a  factor,  by  combining  the  factors  a  solution  can 
be  reached  in  terms  of  discriminative  fabric  perceptions. 

If  we  then  rework  them  in  terms  of  words  associated  with  the  ‘wearing’  of  these  textures,  i.e.  the  emotional  self: 


Table  6.  Fabric  vs.  Factor  vs.  Self 


Fabric 

Combined  Factors 

Emotion/Self 

Denim 

Energised  -  Mossy 

Happy,  young,  confident,  impulsive 

Lycra 

Energised  -  Mossy 

Happy,  young,  impulsive,  nervous 

Fleece 

Sedate  Wannth  -  Protective 

Happy,  content,  satisfied,  reserved, 

Corduroy 

Sedate  Warmth  -  Protective 

Happy ,  content,  reflective 

Leather 

Male -Animal 

Fear,  Anger,  Surprise,  passionate,  impulsive,  strength 

Velvet 

Opulent  Poise  -  Silky 

Happy,  loving,  sensual,  sleek,  sophisticated,  extravagant, 
reflective,  spiritual 

Satin  Silk 

Opulent  Poise  -  Silky 

Happy,  loving,  sensual,  reflective,  sleek,  sophisticated,  open 

Irish  Linen 

Opulent  Poise  -  Silky 

Happy,  sensitive,  reflective,  spiritual,  moralistic 

Tweed 

Faded  Familiarity  -  Coarse 

Content,  reserved,  isolated,  moralistic,  eccentric 

Lace 

Faded  Familiarity  -  Coarse 

Sad,  sensitive/sentimental,  uneasy,  shy,  insecure,  confused, 
isolated,  nervous 

In  the  visual  response  given  by  students,  outlined  below  in  two  examples,  the  relationship  between  fabric  and  the  self, 
as  above,  is  evident. 


3.  Visual  Response 

Mood  Board  Exercise.  Based  on  the  above  experience,  students  were  given  a  brief  to  select  their  own  choice  of 
fabric  to  work  from  and  create  a  visual  response  to.  The  mood  boards  were  to  selectively  reflect  some  of  their  responses 
to  the  physical  evaluation  of  the  fabric  they  selected.  (These  research/ideas  boards  are  used  by  designers  to  develop 
design  philosophies.)  On  completion  they  were  then  asked  to  label  their  boards  with  a  word  or  words  that  encapsulate  a 
visual  conclusion.  It  should  be  noted  that  a  majority  of  the  students  were  previously  unfamiliar  with  using  or  working 
with  computers  prior  to  this  module.  They  were  given  2  weeks  (6  hour  total  module  time)  to  complete  it.  It  shows  a 
variable  response  to  different  fabric  qualities  and  on  the  basis  of  individual  choice,  choices  they  made  based  on 
curiosity  or  that  they  may  relate  to.  It  proved  a  positive  experience  in  terms  of  introducing  virtual  technologies  to  a 
group  that  work  in  a  particularly  tactile  or  hands-on  manner.  Two  examples  are  given  below: 


Figure  6.  Childhood  Summers 

Soft,  light,  cool,  breathable,  childhood,  summer,  fun, 
nostalgic,  sweets,  pretty,  feminine,  light,  home-made, 
homely,  mothers,  chintz,  comfortable  clothes,  1950' s, 
kitsch,  afternoon  tea,  games,  ‘my  old  dresses’. 


Figure  7.  Slithery  Lizzie 

Cold,  tightly  woven,  smooth,  small  grooves,  slithery,  wet 
and  slippy,  skin  shiny,  leather  boots,  frogs,  lizards, 
crocodiles,  sleek,  sexy,  punk,  prostitute. 
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In  the  first  story,  ‘Chiklhootl  Summers’,  the  colour  (pink  and  while),  fabric  type  (100%  Cotton)  and  weight  (light)  ot 
the  fabric,  builds  a  visual  story  and  a  leproscntatum  of  a  friendly  character  and  whimsical  environment.  She  looks 
delighted  and  in  her  light-henrled  garden  of  innocence,  domesticated  and  moralistic  nostalgia.  In  the  story  of  ‘Slithery 
Lizzie’  the  slippery  coated  rigid  fabric  and  texture  has  been  used  to  create  a  possibly  hostile  character  of  the  night  that 
is  as  manipulative  and  corrupt,  evil  or  just  as  strong,  sassy,  and  free  as  a  wild  ctocodilc  ot  snake .  d  ct  she  is  also  timid, 
mischievous  and  cute  as  a  small  frog. 

An  example  of  this  seasons  (sunnier)  visual  and  oral  texture  messaging  is  as  follows'.  Who  can  resist  clothes  that  look  like 
they’ve  been  stolen  from  your  little  sister’s  wardrobe?  ...  What  grown-up  girl  doesn’t  yearn  to  don  dreamy  dressing- 
up-box  clothes  that  their  great  grandmothers  might  once  have  worn  ....  What  did  you  do  in  the  war  mummy?  something 
more  whimsical  for  those  ‘let’s  pretend'  moments  ...  Right  now,  fashion  is  a  real  giggle'.  Sec  Figure  8,  below 


Figure  8.  ‘What  did  you  do  in  the  \va 
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4.  Handle 

In  terms  of  the  ergonomic  criteria  for  a  suitable  haptic  device,  four  handle  methods  shown  below  show  fabtic  piopcttics 
evaluated  by  handle  type.  As  suggested  earlier,  the  multiple  finger  pinch  and  the  touch-stroke  are  the  most  relevant. 
However,  using  just  tire  index  finger  has  also  proved  acceptable. 


Figure  9. 


Handle  Techniques 

1.  Touch-Stroke 


4.  Two  Handed  Rotation 


Handle  Technique 

Properties  Evaluated 

i 

Touch- Stroke 

Surface  quality  (texture),  temperature 

2 

Rotating  Cupped 

Action 

Stiffness,  weight,  temperature,  comfort, 
overall  texture,  creasing 

3 

Multiple  Finger  Pinch:  Rotating 
between  (tic  1  ingots  action  with 
one  hand 

(thumb  and!  or  2  lingers) 

Texture,  stiffness,  temperature,  i'ahric 
structure,  both  sides  of  a  fabric,  friction, 
stretch  (force-feedback) 

4 

Two  Hum  loci  Rdiatitin  Action 

Stretch,  sheerness 

5.  Conclusions 

This  research  introduces  the  delightful  aspects  of  fabric  perception  through  an  examination  of  visual,  touch  and 
cmotional/cognitive/mood  impact.  The  evaluation  ot  this  multi-modal,  cognitive  and  emotional  experience,  examined 
together,  offers  a  reality  model  in  the  development  of  a  vocabulary,  oral  and  aesthetic  texture  ,  that  could  be 
implemented  in  the  development  of  control  variables  for  interactive  communication  within  a  virtual  environment. 
Combined  with  previous  work  where  evaluation  was  based  on  existing  sensory  evaluation  techniques.  1141  1  both 
sealable  and  detailed  analysis  of  fabric  surface  and  its  properties  coupled  with  humane  assessment  of  fabric  perceptions 
can  be  established  that  can  Vie  used  to  refine  haptic  technology  for  the  fashion  and  textiles  and  clothing  arena. 

‘The  hypothesis  is  developed  that  brains  are  designed  around  reward  and  punishment-evaluation  systems,  because  this 
is  the  wav  that  genes  can  build  a  complex  system  that  will  produce  appropriate  but  flexible  behaviour  to  increase 
fitness  ...  The  reason  that  both  emotion  and  motivation  are  treated  [considered  together]  is  that  both  involve  rewards 
and  punishments  as  the  fundamental  solution  of  the  brain  for  interfacing  sensory  systems  to  action-selection  and 
execution  systems. The  amygdala  and  the  orbi I o frontal  cortex  arc  areas  in  the  brain  associated  with  emotion  and 
motivation,  stimulated  by  for  example  sensory  stimuli  and  cognitive  processing  of  language,  memory  and  association. 
Functions  of  emotion  arc  involved  in  the  following  responses:  autonomic,  behavioural,  motivational,  communication, 
social  bonding,  cognitive  evaluation  —  events  and  memories,  and  in  the  storage  ol  memories.  These  responses  aie 
associated  with  clothing  as  with  most  stimuli,  however,  with  the  wearing  and  selection  of  clothing,  a  closer 
understanding  to  the  relationship  ol  the  sell  can  be  examined. 

The  next  phase  of  this  research  that  will  tlieicfore  involve  evaluating  and  measuring  psychological  and  neural  responses 
to  clothing  and  fashion,  i.e.  fabtic  in  context.  It  is  anticipated  that  the  research,  as  well  as  providing  the  necessary  data 
for  translation  in  a  virtual  sense,  could  also  provide  insight  into  medical  disorders  associated  with  these  systems  in  the 
brain. 
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force  magnitude  paralleled  the  actual  force. 

Although  the  fit  was  not  perfect,  these  preliminary 
results  clearly  validate  the  use  of  bio-mechanical  models  of 
the  arm  to  analyze  the  interaction  force  recorded  during 
imposed  motions.  Future  work  with  a  larger  set  of  stimuli 
might  be  necessary  to  assess  whether  more  complex  models 
should  be  considered. 
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Abstract 

Computer  haptics  is  increasingly  becoming  available 
for  use  within  human  machine  dialogues  as  display 
capabilities  improve  and  their  cost  falls.  The  next 
obvious  development  is  for  haptics  to  be  included  within 
multimodal  applications  running  over  networks  using 
Internet  protocols  (IP).  Clearly,  in  order  to  engineer 
networks  to  carry  haptic  information  in  an  efficient  and 
cost-effective  manner,  it  is  necessaiy  to  quantify  how  the 
characteristics  of  the  transmission  path  impact  on  the 
user  perceptions  of  haptic  image  quality. 

This  paper  will  report  the  results  of  experimentation 
with  our  haptic  network  simulation  into  the  impact 
network  latency  on  the  user  perception  for  quality  of 
transmission  of  a  haptic  image.  And,  it  will  briefly 
discuss  the  implications  this  research  may  have  for  the 
transmission  of  haptics  over  IP  networks  and  QoS 
measurement. 


1.  Introduction 

Quality  of  service  (QoS)  models  and  assessment 
methods  exist  for  visual  and  auditory  transmission.  These 
will  need  to  be  combined  with  haptics,  and  other  new 
types  of  sensory  information  emerging  onto  public 
networks,  to  provide  QoS  tools  for  assessing  the 
capability  of  networks  to  carry  multimodal  information. 

For  the  transmission  of  visual  and  auditory  information 
a  key  constraint  that  impacts  on  image  quality  is  the 
bandwidth  available  for  the  transmission.  However,  for 
an  experimental  haptic  transmission  system  we  developed 
-  this  linked  two  Phantom  display  units  via  an  IP  network 
to  enable  two  way  communication  between  them  -  we 
found  bandwidth  to  be  a  minor  issue.  The  major  factor 
that  governed  the  image  quality  in  our  haptic  transmission 
was  the  latency  of  the  network  over  which  it  was  running. 

Latency  is  the  time  taken  within  a  network  for  the 
round  trip  of  a  packet  of  information  to  a  destination  and 
back  to  the  source.  With  our  simple  haptic  transmission 
system  we  saw  instability  and  communication  break  down 
when  network  latency  increased.  To  investigate  this 


further,  and  to  enable  us  to  develop  strategies  to  control  or 
manage  the  impact  of  latency  on  haptic  IP  transmission, 
we  built  a  simple  haptic  network  simulator  that  allowed  us 
to  control  the  latency  within  a  simulated  haptic 
transmission  path. 

One  end  of  our  network  simulation  was  a  virtual 
Phantom  haptic  display  which  could  have  a  virtual  object 
placed  in  front  of  it.  A  real  Phantom  was  then  used  to  feel 
the  remote  virtual  object  through  the  simulated  network 
by  communicating  with  the  virtual  Phantom.  The  latency 
of  the  network  between  the  real  and  virtual  Phantoms  was 
varied  to  investigate  the  impact  on  the  user  perception  of 
the  transmission  quality. 

The  latency  range  used  in  the  experiment  was  0  to  40 
milliseconds  in  10  millisecond  steps.  This  roughly 
covered  the  range  found  when  we  used  the  “Ping” 
network  tool  to  check  the  Internet  latency  from  our 
laboratory  to  a  small  selection  of  UK  Universities  where 
computer  haptics  research  is  currently  taking  place.  Since 
we  may  need  to  engage  in  haptic  communication  over 
these  network  routes  in  future  studies  they  were  used  to 
set  the  latency  range  for  our  experiment. 

2.  Methodology  and  procedure 

An  experimental  control  program  was  written  in 
LabView  and  C++  to  control  the  presentation  of  an  image 
presented  to  a  participant  with  a  Phantom  haptic  display. 
The  image  simulated  an  object  that  had  been  sent  from  a 
remote  Phantom  over  an  Internet  link  with  a  controlled 
level  of  latency  and  a  little  simple  damping.  Each 
participant  carried  out  a  simple  object  discrimination  task. 

For  a  level  of  latency  a  participant  was  asked  to 
determine  if  the  image  being  presented  by  a  Phantom  was 
a  cube  or  sphere.  After  each  object  presentation  the 
subject  adjusted  an  on  screen  cursor  with  a  mouse  to 
provide  an  image  quality  judgement  on  a  rating  scale. 

The  rating  scale  was  one  developed  for  use  in 
transmission  performance  testing  (ITU  1998)  and  has 
been  used  for  on-screen  presentation  in  audiovisual 
quality  assessments  (Hands  2001).  The  participants  were 
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also  asked  to  indicate  whether  the  image  quality  was 
acceptable  before  going  on  to  the  next  object  presentation. 

Each  subject  received  5  levels  of  latency  (0,  10,  20,  30 
and  40  milliseconds)  for  both  a  sphere  and  cube 
recognition  task  and  repeated  5  times.  The  presentation 
order  of  the  treatment  combinations  was  randomised. 

There  were  16  participants  who  were  BT  employees 
working  in  the  Adstral  Park  site  near  Ipswich. 


3.  Results 

The  experimental  control  program  recorded 
measurements  of  the  user  interaction  with  the  virtual 
objects.  This  enabled  both  objective  measures  of  the 
participants  performance  to  be  taken  as  well  as  the  quality 
of  service  ratings  that  the  participant  provided. 

Statistical  testing  of  the  rating  scale  score  with  a  2  way 
repeated  measures  ANOVA  indicated  that  there  was  a 
significant  difference  in  user  perception  of  image  quality 
as  a  result  of  increasing  latency  F(4,  60)  =  34.9,  p<0.0001 
There  was  also  a  difference  between  the  object  shape  F(l, 
15)  =  26.909,  p=0.000I,  and  a  significant  interaction 
between  shape  and  latency  F(4,  60)  =  14.908,  p<0.0001 

The  average  acceptability  score  was  also  tested  with  a 
2  way  repeated  ANOVA.  Again  latency  was  statistically 
significant  (F(4,  60)  =  21.164,  p<0.0001),  but  there  was 
not  a  significant  result  for  object  shape  or  the  interaction. 

Testing  the  participant’s  recognition  times  did  not 
provide  any  statistically  significant  results.  Neither  was 
their  any  statistically  significant  differences  in  the 
accuracy  with  which  participants  were  able  to  identify 
whether  an  object  was  a  sphere  or  a  cube. 


4.  Discussion 

As  latency  increased  in  this  experiment  the  users 
perception  of  the  quality  of  the  haptic  image  was  reduced. 
This  was  not  a  simple  linear  relationship  with  increasing 
latency.  The  reduction  in  perceived  quality  interacted 
with  the  type  of  object  being  displayed.  For  the  sphere 
degradation  was  not  as  large  as  for  the  cube  as  the  latency 
increased  see  fig  1. 

Fig.  1  Interaction  Bar  Plot  for  Average  Quality  of  Service 


Clearly,  this  type  of  quality  measurement  is  going  to  be 
task  dependent.  Any  task  used  will  need  to  be 
representative  of  the  eventual  use  of  the  transmission 
path. 

Traditionally  in  communications  networks  the  major 
concern  has  been  with  the  bandwidth  limitations.  With 
increasing  use  of  new  forms  of  sensory  information  being 
used  over  networks  it  will  be  necessary  for  network 
providers  to  offer  a  quality  of  service  that  is  both 
economic  and  effective  for  the  consumer.  If  latency 
cannot  be  avoided  then  it  will  be  necessary  to  develop 
methods  for  transmitting  haptics  that  are  robust  with 
respect  to  latency  (Wilson  et  al  1999). 


5.  Conclusions 

A  better  understanding  of  human  communication  with 
computer  haptic  simulations,  and  interactions  with  other 
sensory  modalities,  will  indicate  where  it  may  be  possible 
to  make  engineering  trade  offs  to  manage  down  or  remove 
the  impact  of  latency  on  the  customers  perceptions.  There 
is  much  work  that  needs  to  be  done  to  enable  robust 
reliable  real  time  haptic  communication  to  take  place  over 
public  IP  networks  with  a  reasonable  quality  of 
performance  when  latency  is  present. 
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